Oct 30 21:36:52.804998 ip-10-0-137-181 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 30 21:36:52.805011 ip-10-0-137-181 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 30 21:36:52.805021 ip-10-0-137-181 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 30 21:36:52.805341 ip-10-0-137-181 systemd[1]: Failed to start Kubernetes Kubelet. Oct 30 21:37:02.970998 ip-10-0-137-181 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Oct 30 21:37:02.971015 ip-10-0-137-181 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 5913ca457067416bbca11d0354b90a25 -- Oct 30 21:39:21.671899 ip-10-0-137-181 systemd[1]: Starting Kubernetes Kubelet... Oct 30 21:39:22.073359 ip-10-0-137-181 kubenswrapper[2574]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:39:22.073359 ip-10-0-137-181 kubenswrapper[2574]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 21:39:22.073359 ip-10-0-137-181 kubenswrapper[2574]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:39:22.073359 ip-10-0-137-181 kubenswrapper[2574]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:39:22.076489 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076367 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:39:22.076489 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076488 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:39:22.076489 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076492 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:39:22.076581 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076496 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:39:22.076581 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076499 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:39:22.076581 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076502 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:39:22.076581 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076504 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:39:22.076581 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076507 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:39:22.076581 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076510 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:39:22.076581 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076512 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:39:22.076581 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076515 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:39:22.076581 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076518 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:39:22.076581 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076520 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:39:22.076581 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076523 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:39:22.076581 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076525 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:39:22.076581 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076528 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:39:22.076581 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076530 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:39:22.076581 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076533 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:39:22.076581 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076536 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:39:22.076581 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076539 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:39:22.076581 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076541 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:39:22.076581 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076544 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:39:22.077011 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076546 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:39:22.077011 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076549 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:39:22.077011 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076558 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:39:22.077011 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076561 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:39:22.077011 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076564 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:39:22.077011 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076566 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:39:22.077011 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076569 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:39:22.077011 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076572 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:39:22.077011 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076575 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:39:22.077011 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076577 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:39:22.077011 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076579 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:39:22.077011 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076582 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:39:22.077011 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076585 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:39:22.077011 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076588 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:39:22.077011 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076591 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:39:22.077011 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076594 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:39:22.077011 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076596 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:39:22.077011 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076599 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:39:22.077011 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076603 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:39:22.077463 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076607 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:39:22.077463 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076610 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:39:22.077463 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076613 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:39:22.077463 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076615 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:39:22.077463 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076618 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:39:22.077463 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076620 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:39:22.077463 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076623 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:39:22.077463 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076625 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:39:22.077463 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076628 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:39:22.077463 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076630 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:39:22.077463 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076632 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:39:22.077463 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076635 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:39:22.077463 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076637 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:39:22.077463 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076640 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:39:22.077463 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076644 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:39:22.077463 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076647 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:39:22.077463 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076649 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:39:22.077463 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076652 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:39:22.077463 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076655 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:39:22.077463 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076657 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:39:22.077945 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076660 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:39:22.077945 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076662 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:39:22.077945 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076665 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:39:22.077945 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076668 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:39:22.077945 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076670 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:39:22.077945 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076674 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:39:22.077945 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076676 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:39:22.077945 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076679 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:39:22.077945 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076682 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:39:22.077945 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076684 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:39:22.077945 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076686 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:39:22.077945 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076689 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:39:22.077945 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076692 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:39:22.077945 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076694 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:39:22.077945 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076697 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:39:22.077945 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076699 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:39:22.077945 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076702 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:39:22.077945 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076705 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:39:22.077945 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076708 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:39:22.077945 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076710 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:39:22.078473 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076713 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:39:22.078473 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076715 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:39:22.078473 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076717 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:39:22.078473 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076721 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:39:22.078473 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076724 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:39:22.078473 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076727 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:39:22.078473 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076730 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:39:22.078473 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076733 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:39:22.078473 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076736 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:39:22.078473 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076738 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:39:22.078473 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076741 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:39:22.078473 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076743 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:39:22.078473 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076748 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:39:22.078473 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076750 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:39:22.078473 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076753 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:39:22.078473 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076755 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:39:22.078473 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076758 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:39:22.078473 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076760 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:39:22.078473 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076763 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:39:22.078951 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076765 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:39:22.078951 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076767 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:39:22.078951 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076770 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:39:22.078951 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076772 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:39:22.078951 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076774 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:39:22.078951 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.076777 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:39:22.078951 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077570 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:39:22.078951 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077577 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:39:22.078951 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077580 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:39:22.078951 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077583 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:39:22.078951 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077586 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:39:22.078951 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077589 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:39:22.078951 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077591 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:39:22.078951 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077594 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:39:22.078951 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077597 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:39:22.078951 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077599 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:39:22.078951 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077602 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:39:22.078951 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077604 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:39:22.078951 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077608 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:39:22.078951 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077612 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:39:22.079456 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077615 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:39:22.079456 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077617 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:39:22.079456 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077620 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:39:22.079456 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077623 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:39:22.079456 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077625 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:39:22.079456 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077628 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:39:22.079456 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077630 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:39:22.079456 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077633 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:39:22.079456 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077635 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:39:22.079456 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077638 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:39:22.079456 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077640 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:39:22.079456 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077643 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:39:22.079456 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077645 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:39:22.079456 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077648 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:39:22.079456 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077650 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:39:22.079456 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077653 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:39:22.079456 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077655 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:39:22.079456 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077657 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:39:22.079456 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077667 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:39:22.079904 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077672 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:39:22.079904 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077675 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:39:22.079904 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077678 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:39:22.079904 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077681 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:39:22.079904 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077684 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:39:22.079904 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077688 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:39:22.079904 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077691 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:39:22.079904 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077693 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:39:22.079904 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077696 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:39:22.079904 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077698 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:39:22.079904 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077701 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:39:22.079904 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077704 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:39:22.079904 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077707 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:39:22.079904 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077710 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:39:22.079904 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077712 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:39:22.079904 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077715 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:39:22.079904 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077717 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:39:22.079904 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077720 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:39:22.079904 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077722 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:39:22.079904 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077725 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:39:22.080392 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077727 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:39:22.080392 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077730 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:39:22.080392 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077733 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:39:22.080392 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077735 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:39:22.080392 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077737 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:39:22.080392 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077740 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:39:22.080392 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077742 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:39:22.080392 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077745 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:39:22.080392 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077747 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:39:22.080392 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077750 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:39:22.080392 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077752 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:39:22.080392 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077754 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:39:22.080392 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077757 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:39:22.080392 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077759 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:39:22.080392 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077762 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:39:22.080392 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077764 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:39:22.080392 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077766 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:39:22.080392 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077769 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:39:22.080392 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077772 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:39:22.080875 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077774 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:39:22.080875 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077777 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:39:22.080875 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077780 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:39:22.080875 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077782 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:39:22.080875 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077786 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:39:22.080875 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077788 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:39:22.080875 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077791 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:39:22.080875 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077794 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:39:22.080875 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077796 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:39:22.080875 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077798 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:39:22.080875 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077801 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:39:22.080875 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077803 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:39:22.080875 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077805 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:39:22.080875 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077808 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:39:22.080875 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077810 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:39:22.080875 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077813 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:39:22.080875 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077815 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:39:22.080875 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077817 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:39:22.080875 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077820 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:39:22.081351 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077822 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:39:22.081351 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077825 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:39:22.081351 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077827 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:39:22.081351 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077830 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:39:22.081351 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077832 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:39:22.081351 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077835 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:39:22.081351 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077837 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:39:22.081351 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077840 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:39:22.081351 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077842 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:39:22.081351 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077844 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:39:22.081351 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077847 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:39:22.081351 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077849 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:39:22.081351 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077852 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:39:22.081351 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077863 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:39:22.081351 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.077868 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:39:22.081351 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078340 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:39:22.081351 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078345 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:39:22.081351 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078352 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:39:22.081351 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078355 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:39:22.081829 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078357 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:39:22.081829 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078360 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:39:22.081829 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078362 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:39:22.081829 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078365 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:39:22.081829 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078368 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:39:22.081829 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078370 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:39:22.081829 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078373 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:39:22.081829 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078376 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:39:22.081829 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078379 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:39:22.081829 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078400 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:39:22.081829 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078405 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:39:22.081829 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078409 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:39:22.081829 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078413 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:39:22.081829 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078416 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:39:22.081829 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078419 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:39:22.081829 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078421 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:39:22.081829 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078424 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:39:22.081829 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078426 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:39:22.081829 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078429 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:39:22.082275 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078432 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:39:22.082275 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078434 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:39:22.082275 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078436 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:39:22.082275 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078439 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:39:22.082275 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078441 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:39:22.082275 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078444 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:39:22.082275 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078446 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:39:22.082275 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078450 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:39:22.082275 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078455 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:39:22.082275 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078458 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:39:22.082275 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078461 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:39:22.082275 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078463 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:39:22.082275 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078470 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:39:22.082275 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078473 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:39:22.082275 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078476 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:39:22.082275 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078479 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:39:22.082275 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078481 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:39:22.082275 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078484 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:39:22.082275 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078486 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:39:22.082275 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078489 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:39:22.082792 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078492 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:39:22.082792 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078495 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:39:22.082792 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078499 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:39:22.082792 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078503 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:39:22.082792 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078506 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:39:22.082792 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078509 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:39:22.082792 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078511 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:39:22.082792 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078514 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:39:22.082792 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078516 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:39:22.082792 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078518 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:39:22.082792 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078521 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:39:22.082792 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078523 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:39:22.082792 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078526 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:39:22.082792 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078528 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:39:22.082792 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078531 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:39:22.082792 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078533 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:39:22.082792 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078536 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:39:22.082792 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078538 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:39:22.082792 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078541 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:39:22.083226 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078543 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:39:22.083226 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078547 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:39:22.083226 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078550 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:39:22.083226 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078552 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:39:22.083226 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078555 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:39:22.083226 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078558 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:39:22.083226 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078560 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:39:22.083226 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078563 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:39:22.083226 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078565 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:39:22.083226 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078568 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:39:22.083226 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078570 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:39:22.083226 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078572 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:39:22.083226 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078575 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:39:22.083226 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078578 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:39:22.083226 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078580 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:39:22.083226 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078583 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:39:22.083226 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078585 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:39:22.083226 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078588 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:39:22.083226 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078590 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:39:22.083226 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078593 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:39:22.083745 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078595 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:39:22.083745 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078597 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:39:22.083745 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078600 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:39:22.083745 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078602 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:39:22.083745 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078605 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:39:22.083745 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078607 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:39:22.083745 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078609 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:39:22.083745 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078612 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:39:22.083745 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078614 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:39:22.083745 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078616 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:39:22.083745 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078619 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:39:22.083745 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078621 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:39:22.083745 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078624 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:39:22.083745 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078627 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:39:22.083745 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078630 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:39:22.083745 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078633 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:39:22.083745 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078635 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:39:22.083745 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078638 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:39:22.083745 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078641 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:39:22.083745 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078643 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:39:22.084231 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078646 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:39:22.084231 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078648 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:39:22.084231 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078650 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:39:22.084231 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.078654 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:39:22.084231 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.078748 2574 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 21:39:22.084231 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.078756 2574 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 21:39:22.084231 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.078761 2574 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 21:39:22.084231 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.078766 2574 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 21:39:22.084231 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.078772 2574 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 21:39:22.084231 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.078778 2574 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 21:39:22.084231 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.078784 2574 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 21:39:22.084231 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.078790 2574 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 21:39:22.084231 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.078795 2574 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 21:39:22.084231 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.078798 2574 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 21:39:22.084231 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.078802 2574 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 21:39:22.084231 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.078805 2574 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 21:39:22.084231 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.078808 2574 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 21:39:22.084231 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.078811 2574 flags.go:64] FLAG: --cgroup-root="" Oct 30 21:39:22.084231 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.078813 2574 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 21:39:22.084231 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.078816 2574 flags.go:64] FLAG: --client-ca-file="" Oct 30 21:39:22.084231 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.078819 2574 flags.go:64] FLAG: --cloud-provider="external" Oct 30 21:39:22.084231 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.078822 2574 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 21:39:22.084231 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.078827 2574 flags.go:64] FLAG: --cluster-domain="" Oct 30 21:39:22.084231 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.078830 2574 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 21:39:22.084836 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.078833 2574 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 21:39:22.084836 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.078838 2574 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 21:39:22.084836 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.078841 2574 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 21:39:22.084836 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.078846 2574 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 21:39:22.084836 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.078849 2574 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 21:39:22.084836 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.078852 2574 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 21:39:22.084836 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.078855 2574 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 21:39:22.084836 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.078858 2574 flags.go:64] FLAG: --contention-profiling="false" Oct 30 21:39:22.084836 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.078861 2574 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 21:39:22.084836 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.078863 2574 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 21:39:22.084836 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.078866 2574 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 21:39:22.084836 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.078869 2574 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 21:39:22.084836 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.078874 2574 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 21:39:22.084836 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.078877 2574 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 21:39:22.084836 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.078879 2574 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 21:39:22.084836 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.078882 2574 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 21:39:22.084836 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.078885 2574 flags.go:64] FLAG: --enable-server="true" Oct 30 21:39:22.084836 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.078888 2574 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 21:39:22.084836 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.078891 2574 flags.go:64] FLAG: --event-burst="100" Oct 30 21:39:22.084836 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.078894 2574 flags.go:64] FLAG: --event-qps="50" Oct 30 21:39:22.084836 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.078897 2574 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 21:39:22.084836 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.078900 2574 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 21:39:22.084836 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.078904 2574 flags.go:64] FLAG: --eviction-hard="" Oct 30 21:39:22.084836 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.078909 2574 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 21:39:22.085459 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.078912 2574 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 21:39:22.085459 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.078915 2574 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 21:39:22.085459 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.078918 2574 flags.go:64] FLAG: --eviction-soft="" Oct 30 21:39:22.085459 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.078921 2574 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 21:39:22.085459 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.078923 2574 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 21:39:22.085459 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.078926 2574 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 21:39:22.085459 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.078929 2574 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 21:39:22.085459 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.078932 2574 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 21:39:22.085459 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.078935 2574 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 21:39:22.085459 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.078940 2574 flags.go:64] FLAG: --feature-gates="" Oct 30 21:39:22.085459 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.078947 2574 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 21:39:22.085459 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.078950 2574 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 21:39:22.085459 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.078954 2574 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 21:39:22.085459 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.078957 2574 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 21:39:22.085459 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.078960 2574 flags.go:64] FLAG: --healthz-port="10248" Oct 30 21:39:22.085459 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.078963 2574 flags.go:64] FLAG: --help="false" Oct 30 21:39:22.085459 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.078966 2574 flags.go:64] FLAG: --hostname-override="ip-10-0-137-181.us-east-2.compute.internal" Oct 30 21:39:22.085459 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.078969 2574 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 21:39:22.085459 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.078972 2574 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 21:39:22.085459 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.078975 2574 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 21:39:22.085459 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.078978 2574 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 21:39:22.085459 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.078982 2574 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 21:39:22.085459 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.078984 2574 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 21:39:22.086056 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.078987 2574 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 21:39:22.086056 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.078990 2574 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 21:39:22.086056 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.078993 2574 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 21:39:22.086056 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.078995 2574 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 21:39:22.086056 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.078998 2574 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 21:39:22.086056 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.079001 2574 flags.go:64] FLAG: --kube-reserved="" Oct 30 21:39:22.086056 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.079004 2574 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 21:39:22.086056 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.079006 2574 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 21:39:22.086056 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.079009 2574 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 21:39:22.086056 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.079012 2574 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 21:39:22.086056 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.079015 2574 flags.go:64] FLAG: --lock-file="" Oct 30 21:39:22.086056 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.079017 2574 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 21:39:22.086056 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.079020 2574 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 21:39:22.086056 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.079023 2574 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 21:39:22.086056 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.079030 2574 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 21:39:22.086056 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.079033 2574 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 21:39:22.086056 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.079036 2574 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 21:39:22.086056 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.079039 2574 flags.go:64] FLAG: --logging-format="text" Oct 30 21:39:22.086056 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.079043 2574 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 21:39:22.086056 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.079046 2574 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 21:39:22.086056 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.079049 2574 flags.go:64] FLAG: --manifest-url="" Oct 30 21:39:22.086056 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.079052 2574 flags.go:64] FLAG: --manifest-url-header="" Oct 30 21:39:22.086056 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.079056 2574 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 21:39:22.086056 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.079060 2574 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 21:39:22.086056 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.079063 2574 flags.go:64] FLAG: --max-pods="110" Oct 30 21:39:22.086773 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.079066 2574 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 21:39:22.086773 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.079069 2574 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 21:39:22.086773 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.079072 2574 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 21:39:22.086773 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.079075 2574 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 21:39:22.086773 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.079077 2574 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 21:39:22.086773 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.079080 2574 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 21:39:22.086773 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.079083 2574 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 21:39:22.086773 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.079091 2574 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 21:39:22.086773 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.079093 2574 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 21:39:22.086773 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.079096 2574 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 21:39:22.086773 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.079099 2574 flags.go:64] FLAG: --pod-cidr="" Oct 30 21:39:22.086773 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.079102 2574 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 21:39:22.086773 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.079105 2574 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 21:39:22.086773 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.079108 2574 flags.go:64] FLAG: --pods-per-core="0" Oct 30 21:39:22.086773 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.079111 2574 flags.go:64] FLAG: --port="10250" Oct 30 21:39:22.086773 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.079113 2574 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 21:39:22.086773 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.079116 2574 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0aac217f1957b8d85" Oct 30 21:39:22.086773 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.079119 2574 flags.go:64] FLAG: --qos-reserved="" Oct 30 21:39:22.086773 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.079122 2574 flags.go:64] FLAG: --read-only-port="10255" Oct 30 21:39:22.086773 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.079125 2574 flags.go:64] FLAG: --register-node="true" Oct 30 21:39:22.086773 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.079128 2574 flags.go:64] FLAG: --register-with-taints="" Oct 30 21:39:22.086773 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.079133 2574 flags.go:64] FLAG: --registry-burst="10" Oct 30 21:39:22.086773 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.079136 2574 flags.go:64] FLAG: --registry-qps="5" Oct 30 21:39:22.086773 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.079139 2574 flags.go:64] FLAG: --reserved-cpus="" Oct 30 21:39:22.086773 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.079142 2574 flags.go:64] FLAG: --reserved-memory="" Oct 30 21:39:22.086773 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.079146 2574 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 21:39:22.087495 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.079149 2574 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 21:39:22.087495 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.079152 2574 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 21:39:22.087495 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.079155 2574 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 21:39:22.087495 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.079158 2574 flags.go:64] FLAG: --runonce="false" Oct 30 21:39:22.087495 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.079161 2574 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 21:39:22.087495 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.079164 2574 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 21:39:22.087495 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.079167 2574 flags.go:64] FLAG: --seccomp-default="false" Oct 30 21:39:22.087495 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.079170 2574 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 21:39:22.087495 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.079173 2574 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 21:39:22.087495 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.079176 2574 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 21:39:22.087495 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.079179 2574 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 21:39:22.087495 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.079182 2574 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 21:39:22.087495 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.079184 2574 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 21:39:22.087495 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.079187 2574 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 21:39:22.087495 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.079190 2574 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 21:39:22.087495 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.079193 2574 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 21:39:22.087495 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.079196 2574 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 21:39:22.087495 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.079200 2574 flags.go:64] FLAG: --system-cgroups="" Oct 30 21:39:22.087495 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.079203 2574 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 21:39:22.087495 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.079207 2574 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 21:39:22.087495 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.079210 2574 flags.go:64] FLAG: --tls-cert-file="" Oct 30 21:39:22.087495 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.079213 2574 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 21:39:22.087495 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.079217 2574 flags.go:64] FLAG: --tls-min-version="" Oct 30 21:39:22.087495 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.079220 2574 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 21:39:22.087495 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.079222 2574 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 21:39:22.088313 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.079225 2574 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 21:39:22.088313 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.079228 2574 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 21:39:22.088313 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.079232 2574 flags.go:64] FLAG: --v="2" Oct 30 21:39:22.088313 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.079236 2574 flags.go:64] FLAG: --version="false" Oct 30 21:39:22.088313 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.079240 2574 flags.go:64] FLAG: --vmodule="" Oct 30 21:39:22.088313 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.079244 2574 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 21:39:22.088313 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.079248 2574 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 21:39:22.088313 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079418 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:39:22.088313 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079422 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:39:22.088313 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079425 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:39:22.088313 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079428 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:39:22.088313 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079430 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:39:22.088313 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079440 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:39:22.088313 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079443 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:39:22.088313 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079446 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:39:22.088313 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079448 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:39:22.088313 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079451 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:39:22.088313 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079453 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:39:22.088313 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079456 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:39:22.088313 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079458 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:39:22.088313 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079461 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:39:22.088313 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079464 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:39:22.088942 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079467 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:39:22.088942 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079470 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:39:22.088942 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079472 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:39:22.088942 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079475 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:39:22.088942 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079477 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:39:22.088942 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079480 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:39:22.088942 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079482 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:39:22.088942 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079485 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:39:22.088942 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079487 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:39:22.088942 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079490 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:39:22.088942 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079492 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:39:22.088942 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079495 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:39:22.088942 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079498 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:39:22.088942 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079501 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:39:22.088942 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079504 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:39:22.088942 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079506 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:39:22.088942 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079510 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:39:22.088942 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079512 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:39:22.088942 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079515 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:39:22.088942 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079517 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:39:22.089511 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079520 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:39:22.089511 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079522 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:39:22.089511 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079524 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:39:22.089511 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079532 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:39:22.089511 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079535 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:39:22.089511 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079538 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:39:22.089511 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079540 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:39:22.089511 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079543 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:39:22.089511 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079545 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:39:22.089511 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079548 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:39:22.089511 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079550 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:39:22.089511 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079552 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:39:22.089511 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079555 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:39:22.089511 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079557 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:39:22.089511 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079561 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:39:22.089511 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079565 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:39:22.089511 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079567 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:39:22.089511 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079570 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:39:22.089511 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079572 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:39:22.090033 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079575 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:39:22.090033 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079577 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:39:22.090033 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079580 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:39:22.090033 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079583 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:39:22.090033 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079585 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:39:22.090033 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079589 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:39:22.090033 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079591 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:39:22.090033 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079593 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:39:22.090033 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079596 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:39:22.090033 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079599 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:39:22.090033 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079602 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:39:22.090033 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079605 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:39:22.090033 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079607 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:39:22.090033 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079610 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:39:22.090033 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079612 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:39:22.090033 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079614 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:39:22.090033 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079617 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:39:22.090033 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079620 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:39:22.090033 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079623 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:39:22.090532 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079625 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:39:22.090532 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079628 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:39:22.090532 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079631 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:39:22.090532 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079633 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:39:22.090532 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079635 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:39:22.090532 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079638 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:39:22.090532 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079640 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:39:22.090532 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079642 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:39:22.090532 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079645 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:39:22.090532 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079647 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:39:22.090532 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079650 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:39:22.090532 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079652 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:39:22.090532 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079654 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:39:22.090532 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079657 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:39:22.090532 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079659 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:39:22.090532 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079662 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:39:22.090532 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079664 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:39:22.090532 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079666 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:39:22.090532 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079670 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:39:22.090532 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079674 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:39:22.091021 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079678 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:39:22.091021 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079681 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:39:22.091021 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079685 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:39:22.091021 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079688 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:39:22.091021 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079690 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:39:22.091021 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079693 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:39:22.091021 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079695 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:39:22.091021 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079698 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:39:22.091021 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079700 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:39:22.091021 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079703 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:39:22.091021 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079705 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:39:22.091021 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079708 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:39:22.091021 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079711 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:39:22.091021 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.079736 2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:39:22.091021 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079773 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:39:22.091021 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079778 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:39:22.091414 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079781 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:39:22.091414 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079784 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:39:22.091414 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079787 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:39:22.091414 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079789 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:39:22.091414 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079792 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:39:22.091414 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079794 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:39:22.091414 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079797 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:39:22.091414 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079799 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:39:22.091414 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079802 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:39:22.091414 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079804 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:39:22.091414 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079807 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:39:22.091414 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079809 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:39:22.091414 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079812 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:39:22.091414 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079814 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:39:22.091414 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079818 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:39:22.091414 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079821 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:39:22.091414 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079823 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:39:22.091414 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079826 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:39:22.091414 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079828 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:39:22.091414 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079832 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:39:22.091899 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079835 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:39:22.091899 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079837 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:39:22.091899 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079840 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:39:22.091899 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079842 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:39:22.091899 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079845 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:39:22.091899 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079847 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:39:22.091899 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079849 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:39:22.091899 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079852 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:39:22.091899 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079855 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:39:22.091899 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079857 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:39:22.091899 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079860 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:39:22.091899 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079864 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:39:22.091899 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079868 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:39:22.091899 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079871 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:39:22.091899 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079873 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:39:22.091899 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079876 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:39:22.091899 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079878 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:39:22.091899 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079881 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:39:22.091899 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079883 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:39:22.092344 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079885 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:39:22.092344 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079888 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:39:22.092344 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079890 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:39:22.092344 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079893 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:39:22.092344 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079895 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:39:22.092344 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079897 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:39:22.092344 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079900 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:39:22.092344 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079904 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:39:22.092344 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079906 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:39:22.092344 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079909 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:39:22.092344 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079911 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:39:22.092344 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079914 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:39:22.092344 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079917 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:39:22.092344 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079920 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:39:22.092344 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079923 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:39:22.092344 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079925 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:39:22.092344 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079928 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:39:22.092344 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079930 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:39:22.092344 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079933 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:39:22.092816 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079935 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:39:22.092816 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079938 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:39:22.092816 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079940 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:39:22.092816 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079943 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:39:22.092816 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079946 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:39:22.092816 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079948 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:39:22.092816 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079950 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:39:22.092816 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079953 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:39:22.092816 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079955 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:39:22.092816 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079958 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:39:22.092816 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079960 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:39:22.092816 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079962 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:39:22.092816 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079965 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:39:22.092816 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079967 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:39:22.092816 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079970 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:39:22.092816 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079973 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:39:22.092816 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079977 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:39:22.092816 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079981 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:39:22.092816 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079983 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:39:22.093268 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079986 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:39:22.093268 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079990 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:39:22.093268 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079992 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:39:22.093268 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079995 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:39:22.093268 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.079997 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:39:22.093268 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.080001 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:39:22.093268 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.080004 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:39:22.093268 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.080006 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:39:22.093268 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.080009 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:39:22.093268 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.080011 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:39:22.093268 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.080014 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:39:22.093268 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.080016 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:39:22.093268 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.080019 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:39:22.093268 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.080021 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:39:22.093268 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.080024 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:39:22.093268 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.080026 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:39:22.093268 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.080028 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:39:22.093268 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.080031 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:39:22.093268 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.080034 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:39:22.093268 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.080037 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:39:22.093811 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.080039 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:39:22.093811 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.080042 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:39:22.093811 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.080044 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:39:22.093811 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.080046 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:39:22.093811 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.080049 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:39:22.093811 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.080051 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:39:22.093811 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.080054 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:39:22.093811 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.087478 2574 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 21:39:22.093811 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.087494 2574 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 21:39:22.093811 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.087508 2574 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 21:39:22.093811 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.087513 2574 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 21:39:22.093811 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087569 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:39:22.093811 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087574 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:39:22.093811 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087577 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:39:22.093811 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087580 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:39:22.093811 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087583 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:39:22.093811 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087586 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:39:22.093811 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087589 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:39:22.093811 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087591 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:39:22.093811 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087594 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:39:22.094315 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087596 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:39:22.094315 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087599 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:39:22.094315 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087601 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:39:22.094315 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087604 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:39:22.094315 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087607 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:39:22.094315 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087610 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:39:22.094315 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087612 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:39:22.094315 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087615 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:39:22.094315 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087617 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:39:22.094315 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087619 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:39:22.094315 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087622 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:39:22.094315 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087624 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:39:22.094315 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087627 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:39:22.094315 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087630 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:39:22.094315 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087632 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:39:22.094315 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087635 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:39:22.094315 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087637 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:39:22.094315 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087640 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:39:22.094315 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087642 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:39:22.094779 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087646 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:39:22.094779 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087652 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:39:22.094779 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087656 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:39:22.094779 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087659 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:39:22.094779 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087662 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:39:22.094779 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087664 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:39:22.094779 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087667 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:39:22.094779 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087670 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:39:22.094779 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087672 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:39:22.094779 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087675 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:39:22.094779 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087677 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:39:22.094779 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087680 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:39:22.094779 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087682 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:39:22.094779 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087685 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:39:22.094779 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087687 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:39:22.094779 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087689 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:39:22.094779 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087692 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:39:22.094779 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087694 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:39:22.094779 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087697 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:39:22.094779 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087700 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:39:22.095244 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087702 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:39:22.095244 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087705 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:39:22.095244 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087707 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:39:22.095244 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087710 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:39:22.095244 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087712 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:39:22.095244 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087715 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:39:22.095244 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087717 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:39:22.095244 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087719 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:39:22.095244 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087722 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:39:22.095244 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087724 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:39:22.095244 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087727 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:39:22.095244 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087730 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:39:22.095244 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087734 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:39:22.095244 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087738 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:39:22.095244 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087741 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:39:22.095244 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087744 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:39:22.095244 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087746 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:39:22.095244 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087749 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:39:22.095244 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087751 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:39:22.095789 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087754 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:39:22.095789 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087756 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:39:22.095789 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087758 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:39:22.095789 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087761 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:39:22.095789 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087764 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:39:22.095789 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087766 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:39:22.095789 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087769 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:39:22.095789 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087771 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:39:22.095789 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087773 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:39:22.095789 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087776 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:39:22.095789 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087778 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:39:22.095789 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087781 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:39:22.095789 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087783 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:39:22.095789 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087786 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:39:22.095789 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087788 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:39:22.095789 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087790 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:39:22.095789 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087793 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:39:22.095789 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087795 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:39:22.095789 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087798 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:39:22.096264 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087800 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:39:22.096264 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087803 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:39:22.096264 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087805 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:39:22.096264 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087808 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:39:22.096264 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087810 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:39:22.096264 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087813 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:39:22.096264 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087816 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:39:22.096264 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087818 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:39:22.096264 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087821 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:39:22.096264 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087824 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:39:22.096264 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087827 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:39:22.096264 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087829 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:39:22.096264 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087832 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:39:22.096264 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087834 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:39:22.096264 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087837 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:39:22.096264 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087839 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:39:22.096264 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087842 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:39:22.096264 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087844 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:39:22.096264 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087847 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:39:22.096755 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.087849 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:39:22.096755 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.087875 2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:39:22.096755 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088011 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:39:22.096755 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088015 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:39:22.096755 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088018 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:39:22.096755 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088021 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:39:22.096755 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088023 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:39:22.096755 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088026 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:39:22.096755 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088029 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:39:22.096755 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088031 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:39:22.096755 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088034 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:39:22.096755 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088036 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:39:22.096755 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088039 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:39:22.096755 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088041 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:39:22.096755 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088044 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:39:22.096755 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088046 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:39:22.097255 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088049 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:39:22.097255 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088051 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:39:22.097255 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088054 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:39:22.097255 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088056 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:39:22.097255 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088059 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:39:22.097255 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088062 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:39:22.097255 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088064 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:39:22.097255 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088067 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:39:22.097255 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088070 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:39:22.097255 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088072 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:39:22.097255 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088075 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:39:22.097255 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088077 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:39:22.097255 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088080 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:39:22.097255 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088082 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:39:22.097255 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088085 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:39:22.097255 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088087 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:39:22.097255 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088090 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:39:22.097255 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088092 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:39:22.097255 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088095 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:39:22.097255 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088097 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:39:22.097767 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088099 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:39:22.097767 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088102 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:39:22.097767 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088104 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:39:22.097767 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088107 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:39:22.097767 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088111 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:39:22.097767 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088115 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:39:22.097767 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088118 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:39:22.097767 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088121 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:39:22.097767 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088124 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:39:22.097767 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088127 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:39:22.097767 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088130 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:39:22.097767 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088132 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:39:22.097767 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088135 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:39:22.097767 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088137 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:39:22.097767 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088140 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:39:22.097767 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088143 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:39:22.097767 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088145 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:39:22.097767 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088148 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:39:22.097767 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088150 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:39:22.097767 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088153 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:39:22.098234 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088156 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:39:22.098234 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088158 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:39:22.098234 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088161 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:39:22.098234 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088163 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:39:22.098234 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088166 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:39:22.098234 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088168 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:39:22.098234 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088171 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:39:22.098234 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088173 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:39:22.098234 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088176 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:39:22.098234 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088178 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:39:22.098234 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088180 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:39:22.098234 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088183 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:39:22.098234 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088186 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:39:22.098234 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088188 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:39:22.098234 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088191 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:39:22.098234 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088193 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:39:22.098234 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088195 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:39:22.098234 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088198 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:39:22.098234 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088200 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:39:22.098796 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088203 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:39:22.098796 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088205 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:39:22.098796 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088207 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:39:22.098796 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088210 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:39:22.098796 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088213 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:39:22.098796 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088215 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:39:22.098796 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088217 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:39:22.098796 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088220 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:39:22.098796 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088223 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:39:22.098796 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088226 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:39:22.098796 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088228 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:39:22.098796 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088231 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:39:22.098796 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088233 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:39:22.098796 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088236 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:39:22.098796 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088239 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:39:22.098796 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088241 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:39:22.098796 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088243 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:39:22.098796 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088246 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:39:22.098796 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088248 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:39:22.099278 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088251 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:39:22.099278 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088253 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:39:22.099278 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088256 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:39:22.099278 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088258 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:39:22.099278 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088261 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:39:22.099278 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088263 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:39:22.099278 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088266 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:39:22.099278 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088268 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:39:22.099278 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088271 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:39:22.099278 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088273 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:39:22.099278 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088276 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:39:22.099278 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088278 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:39:22.099278 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088280 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:39:22.099278 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088284 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:39:22.099278 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.088306 2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:39:22.099278 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088337 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:39:22.099738 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088341 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:39:22.099738 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088343 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:39:22.099738 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088346 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:39:22.099738 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088348 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:39:22.099738 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088352 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:39:22.099738 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088355 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:39:22.099738 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088357 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:39:22.099738 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088360 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:39:22.099738 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088362 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:39:22.099738 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088365 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:39:22.099738 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088367 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:39:22.099738 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088370 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:39:22.099738 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088373 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:39:22.099738 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088376 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:39:22.099738 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088379 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:39:22.099738 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088394 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:39:22.099738 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088397 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:39:22.099738 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088400 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:39:22.099738 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088402 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:39:22.100197 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088406 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:39:22.100197 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088409 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:39:22.100197 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088412 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:39:22.100197 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088414 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:39:22.100197 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088417 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:39:22.100197 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088420 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:39:22.100197 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088422 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:39:22.100197 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088425 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:39:22.100197 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088427 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:39:22.100197 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088430 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:39:22.100197 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088433 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:39:22.100197 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088436 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:39:22.100197 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088438 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:39:22.100197 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088440 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:39:22.100197 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088443 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:39:22.100197 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088445 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:39:22.100197 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088448 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:39:22.100197 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088450 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:39:22.100197 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088453 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:39:22.100700 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088455 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:39:22.100700 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088458 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:39:22.100700 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088460 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:39:22.100700 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088463 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:39:22.100700 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088465 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:39:22.100700 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088468 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:39:22.100700 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088470 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:39:22.100700 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088473 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:39:22.100700 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088476 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:39:22.100700 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088478 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:39:22.100700 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088481 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:39:22.100700 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088483 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:39:22.100700 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088486 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:39:22.100700 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088488 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:39:22.100700 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088490 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:39:22.100700 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088493 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:39:22.100700 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088496 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:39:22.100700 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088498 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:39:22.100700 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088500 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:39:22.101134 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088503 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:39:22.101134 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088507 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:39:22.101134 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088510 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:39:22.101134 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088513 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:39:22.101134 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088516 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:39:22.101134 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088519 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:39:22.101134 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088522 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:39:22.101134 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088524 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:39:22.101134 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088527 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:39:22.101134 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088529 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:39:22.101134 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088531 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:39:22.101134 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088534 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:39:22.101134 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088536 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:39:22.101134 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088539 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:39:22.101134 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088541 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:39:22.101134 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088544 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:39:22.101134 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088546 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:39:22.101134 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088548 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:39:22.101134 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088551 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:39:22.101134 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088553 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:39:22.101622 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088556 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:39:22.101622 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088558 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:39:22.101622 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088561 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:39:22.101622 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088563 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:39:22.101622 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088565 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:39:22.101622 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088568 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:39:22.101622 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088570 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:39:22.101622 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088572 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:39:22.101622 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088575 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:39:22.101622 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088577 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:39:22.101622 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088580 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:39:22.101622 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088582 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:39:22.101622 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088585 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:39:22.101622 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088587 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:39:22.101622 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088589 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:39:22.101622 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088592 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:39:22.101622 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088594 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:39:22.101622 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088597 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:39:22.101622 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088599 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:39:22.101622 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088602 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:39:22.102098 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088604 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:39:22.102098 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088606 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:39:22.102098 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088609 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:39:22.102098 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088611 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:39:22.102098 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088615 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:39:22.102098 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088619 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:39:22.102098 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088622 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:39:22.102098 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:22.088624 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:39:22.102098 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.088674 2574 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 21:39:22.102098 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.090609 2574 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 21:39:22.102098 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.091482 2574 server.go:1014] "Starting client certificate rotation" Oct 30 21:39:22.102098 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.091570 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 21:39:22.102098 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.091600 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 21:39:22.102098 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.093514 2574 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:39:22.102098 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.099533 2574 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:39:22.110091 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.110073 2574 log.go:25] "Validated CRI v1 runtime API" Oct 30 21:39:22.113527 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.113512 2574 log.go:25] "Validated CRI v1 image API" Oct 30 21:39:22.114325 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.114305 2574 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 21:39:22.120717 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.120698 2574 fs.go:135] Filesystem UUIDs: map[3a52f4fd-9681-4d1a-aea0-b982992c991a:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 9f337de4-a428-4115-ae4a-5f6d7a7095e9:/dev/nvme0n1p4] Oct 30 21:39:22.120780 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.120716 2574 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 21:39:22.121827 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.121811 2574 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 21:39:22.125846 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.125734 2574 manager.go:217] Machine: {Timestamp:2026-10-30 21:39:22.124402247 +0000 UTC m=+0.331981197 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097812 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c000c56026d053f4cbf2191645414 SystemUUID:ec2c000c-5602-6d05-3f4c-bf2191645414 BootID:5913ca45-7067-416b-bca1-1d0354b90a25 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ec:48:68:cd Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ec:48:68:cd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:22:8f:dc:2b:68:70 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 21:39:22.125846 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.125846 2574 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 21:39:22.125966 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.125955 2574 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 21:39:22.128567 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.128546 2574 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 21:39:22.128705 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.128573 2574 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-181.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 21:39:22.128750 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.128708 2574 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 21:39:22.128750 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.128715 2574 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 21:39:22.128804 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.128749 2574 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:39:22.128804 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.128767 2574 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:39:22.129489 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.129480 2574 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 21:39:22.133472 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.133460 2574 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 21:39:22.133589 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.133579 2574 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 21:39:22.133632 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.133612 2574 kubelet.go:493] "Attempting to sync node with API server" Oct 30 21:39:22.133632 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.133622 2574 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 21:39:22.133632 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.133632 2574 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 21:39:22.133749 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.133641 2574 kubelet.go:405] "Adding apiserver pod source" Oct 30 21:39:22.133749 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.133654 2574 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 21:39:22.133984 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.133972 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:39:22.136184 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.136164 2574 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 21:39:22.140624 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.140609 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:39:22.142296 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.142274 2574 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 21:39:22.142404 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.142310 2574 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 21:39:22.142534 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.142520 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 21:39:22.142581 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.142546 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 21:39:22.142581 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.142559 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 21:39:22.142581 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.142568 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 21:39:22.142702 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.142588 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 21:39:22.142702 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.142598 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 21:39:22.143121 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.143106 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 21:39:22.143195 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.143186 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 21:39:22.143259 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.143243 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 21:39:22.143510 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.143497 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 21:39:22.143571 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.143518 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 21:39:22.143571 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.143535 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 21:39:22.144657 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.144644 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 21:39:22.144695 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.144660 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 21:39:22.147732 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.147711 2574 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-181.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 21:39:22.148825 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.148812 2574 server.go:1263] "Started kubelet" Oct 30 21:39:22.148908 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.148880 2574 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 21:39:22.148965 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.148905 2574 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 21:39:22.149017 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.148973 2574 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 21:39:22.149605 ip-10-0-137-181 systemd[1]: Started Kubernetes Kubelet. Oct 30 21:39:22.150230 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.150037 2574 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 21:39:22.151134 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.151119 2574 server.go:317] "Adding debug handlers to kubelet server" Oct 30 21:39:22.151982 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:22.151958 2574 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-181.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 21:39:22.151982 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:22.151965 2574 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 21:39:22.155760 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.155740 2574 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 21:39:22.155760 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.155758 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 21:39:22.155919 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.155770 2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qzt24" Oct 30 21:39:22.156081 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:22.156062 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-181.us-east-2.compute.internal\" not found" Oct 30 21:39:22.156424 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.156404 2574 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 21:39:22.156424 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.156425 2574 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 21:39:22.156552 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.156488 2574 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 21:39:22.156552 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.156533 2574 reconstruct.go:97] "Volume reconstruction finished" Oct 30 21:39:22.156552 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.156542 2574 reconciler.go:29] "Reconciler: start to sync state" Oct 30 21:39:22.157016 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.156997 2574 factory.go:55] Registering systemd factory Oct 30 21:39:22.157093 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.157042 2574 factory.go:223] Registration of the systemd container factory successfully Oct 30 21:39:22.157093 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:22.157044 2574 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 21:39:22.157256 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.157239 2574 factory.go:153] Registering CRI-O factory Oct 30 21:39:22.157256 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.157259 2574 factory.go:223] Registration of the crio container factory successfully Oct 30 21:39:22.157449 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.157314 2574 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 21:39:22.157449 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.157333 2574 factory.go:103] Registering Raw factory Oct 30 21:39:22.157449 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.157348 2574 manager.go:1199] Started watching for new ooms in manager Oct 30 21:39:22.158426 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.158378 2574 manager.go:319] Starting recovery of all containers Oct 30 21:39:22.160369 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:22.159282 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-181.us-east-2.compute.internal.18e36c7d535b3227 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-181.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-181.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 21:39:22.148786727 +0000 UTC m=+0.356365677,LastTimestamp:2026-10-30 21:39:22.148786727 +0000 UTC m=+0.356365677,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-181.us-east-2.compute.internal,}" Oct 30 21:39:22.160507 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:22.160341 2574 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-181.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 21:39:22.164073 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:22.164048 2574 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 21:39:22.169875 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.169853 2574 manager.go:324] Recovery completed Oct 30 21:39:22.170988 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.170966 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qzt24" Oct 30 21:39:22.171736 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:22.171715 2574 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Oct 30 21:39:22.175358 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.175345 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:39:22.177139 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.177123 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-181.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:39:22.177197 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.177150 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-181.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:39:22.177197 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.177161 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-181.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:39:22.177690 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.177678 2574 cpu_manager.go:226] "Starting" policy="none" Oct 30 21:39:22.177690 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.177690 2574 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 21:39:22.177788 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.177705 2574 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 21:39:22.177788 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.177744 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:39:22.178424 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.178409 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-181.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:39:22.178468 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.178435 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-181.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:39:22.178468 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.178447 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-181.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:39:22.179367 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:22.179304 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-181.us-east-2.compute.internal.18e36c7d550bcdb2 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-181.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-181.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-181.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 21:39:22.177138098 +0000 UTC m=+0.384717048,LastTimestamp:2026-10-30 21:39:22.177138098 +0000 UTC m=+0.384717048,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-181.us-east-2.compute.internal,}" Oct 30 21:39:22.179849 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.179838 2574 policy_none.go:50] "Start" Oct 30 21:39:22.179888 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.179853 2574 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 21:39:22.179888 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.179862 2574 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 21:39:22.181869 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.181858 2574 policy_none.go:44] "Start" Oct 30 21:39:22.216089 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.215921 2574 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 21:39:22.219917 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.219905 2574 manager.go:341] "Starting Device Plugin manager" Oct 30 21:39:22.219992 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:22.219929 2574 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 21:39:22.219992 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.219937 2574 server.go:87] "Starting device plugin registration server" Oct 30 21:39:22.220152 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.220141 2574 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 21:39:22.220198 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.220152 2574 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 21:39:22.220238 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.220206 2574 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 21:39:22.220303 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.220285 2574 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 21:39:22.220303 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.220296 2574 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 21:39:22.220841 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:22.220825 2574 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 21:39:22.220944 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:22.220870 2574 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-181.us-east-2.compute.internal\" not found" Oct 30 21:39:22.320503 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.320483 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:39:22.322275 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.321476 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-181.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:39:22.322275 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.321508 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-181.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:39:22.322275 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.321518 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-181.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:39:22.322275 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.321559 2574 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-137-181.us-east-2.compute.internal" Oct 30 21:39:22.329538 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.329496 2574 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-137-181.us-east-2.compute.internal" Oct 30 21:39:22.329538 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:22.329520 2574 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-181.us-east-2.compute.internal\": node \"ip-10-0-137-181.us-east-2.compute.internal\" not found" Oct 30 21:39:22.342178 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:22.342161 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-181.us-east-2.compute.internal\" not found" Oct 30 21:39:22.418471 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.418445 2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 21:39:22.419703 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.419685 2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 21:39:22.419789 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.419711 2574 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 21:39:22.419789 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.419731 2574 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 21:39:22.420359 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:22.419770 2574 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 21:39:22.425739 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.425723 2574 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:39:22.442810 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:22.442786 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-181.us-east-2.compute.internal\" not found" Oct 30 21:39:22.521204 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.521177 2574 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-181.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-181.us-east-2.compute.internal"] Oct 30 21:39:22.521272 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.521243 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:39:22.522110 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.522095 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-181.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:39:22.522165 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.522123 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-181.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:39:22.522165 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.522133 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-181.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:39:22.524216 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.524203 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:39:22.524404 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.524374 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-181.us-east-2.compute.internal" Oct 30 21:39:22.524444 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.524419 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:39:22.524933 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.524912 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-181.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:39:22.524933 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.524924 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-181.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:39:22.525044 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.524951 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-181.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:39:22.525044 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.524978 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-181.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:39:22.525044 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.524951 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-181.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:39:22.525044 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.525033 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-181.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:39:22.526914 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.526897 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-181.us-east-2.compute.internal" Oct 30 21:39:22.527005 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.526922 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:39:22.527558 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.527541 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-181.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:39:22.527644 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.527567 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-181.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:39:22.527644 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.527584 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-181.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:39:22.543335 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:22.543320 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-181.us-east-2.compute.internal\" not found" Oct 30 21:39:22.557724 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:22.557710 2574 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-181.us-east-2.compute.internal\" not found" node="ip-10-0-137-181.us-east-2.compute.internal" Oct 30 21:39:22.558239 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.558223 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/952d6c48a2bc0d00a5a217ec8deb4cd1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-181.us-east-2.compute.internal\" (UID: \"952d6c48a2bc0d00a5a217ec8deb4cd1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-181.us-east-2.compute.internal" Oct 30 21:39:22.558279 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.558247 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/952d6c48a2bc0d00a5a217ec8deb4cd1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-181.us-east-2.compute.internal\" (UID: \"952d6c48a2bc0d00a5a217ec8deb4cd1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-181.us-east-2.compute.internal" Oct 30 21:39:22.558279 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.558265 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5fb540c197b86dcf7fd282ae878b96ce-config\") pod \"kube-apiserver-proxy-ip-10-0-137-181.us-east-2.compute.internal\" (UID: \"5fb540c197b86dcf7fd282ae878b96ce\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-181.us-east-2.compute.internal" Oct 30 21:39:22.560209 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:22.560194 2574 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-181.us-east-2.compute.internal\" not found" node="ip-10-0-137-181.us-east-2.compute.internal" Oct 30 21:39:22.643726 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:22.643700 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-181.us-east-2.compute.internal\" not found" Oct 30 21:39:22.659038 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.659016 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/952d6c48a2bc0d00a5a217ec8deb4cd1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-181.us-east-2.compute.internal\" (UID: \"952d6c48a2bc0d00a5a217ec8deb4cd1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-181.us-east-2.compute.internal" Oct 30 21:39:22.659094 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.659039 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/952d6c48a2bc0d00a5a217ec8deb4cd1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-181.us-east-2.compute.internal\" (UID: \"952d6c48a2bc0d00a5a217ec8deb4cd1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-181.us-east-2.compute.internal" Oct 30 21:39:22.659094 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.659063 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5fb540c197b86dcf7fd282ae878b96ce-config\") pod \"kube-apiserver-proxy-ip-10-0-137-181.us-east-2.compute.internal\" (UID: \"5fb540c197b86dcf7fd282ae878b96ce\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-181.us-east-2.compute.internal" Oct 30 21:39:22.659167 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.659124 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/952d6c48a2bc0d00a5a217ec8deb4cd1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-181.us-east-2.compute.internal\" (UID: \"952d6c48a2bc0d00a5a217ec8deb4cd1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-181.us-east-2.compute.internal" Oct 30 21:39:22.659167 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.659149 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5fb540c197b86dcf7fd282ae878b96ce-config\") pod \"kube-apiserver-proxy-ip-10-0-137-181.us-east-2.compute.internal\" (UID: \"5fb540c197b86dcf7fd282ae878b96ce\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-181.us-east-2.compute.internal" Oct 30 21:39:22.659167 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.659121 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/952d6c48a2bc0d00a5a217ec8deb4cd1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-181.us-east-2.compute.internal\" (UID: \"952d6c48a2bc0d00a5a217ec8deb4cd1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-181.us-east-2.compute.internal" Oct 30 21:39:22.744209 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:22.744181 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-181.us-east-2.compute.internal\" not found" Oct 30 21:39:22.844767 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:22.844743 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-181.us-east-2.compute.internal\" not found" Oct 30 21:39:22.861128 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.861107 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-181.us-east-2.compute.internal" Oct 30 21:39:22.863199 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:22.863185 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-181.us-east-2.compute.internal" Oct 30 21:39:22.945472 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:22.945396 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-181.us-east-2.compute.internal\" not found" Oct 30 21:39:23.045893 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:23.045868 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-181.us-east-2.compute.internal\" not found" Oct 30 21:39:23.091477 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:23.091457 2574 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 21:39:23.091888 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:23.091576 2574 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 21:39:23.146505 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:23.146485 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-181.us-east-2.compute.internal\" not found" Oct 30 21:39:23.167898 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:23.167871 2574 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:39:23.176625 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:23.176593 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 21:34:22 +0000 UTC" deadline="2028-07-01 16:13:36.913935277 +0000 UTC" Oct 30 21:39:23.176625 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:23.176621 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14634h34m13.737316544s" Oct 30 21:39:23.225937 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:23.225873 2574 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:39:23.256928 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:23.256909 2574 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-181.us-east-2.compute.internal" Oct 30 21:39:23.269915 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:23.269899 2574 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:39:23.271040 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:23.271028 2574 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-181.us-east-2.compute.internal" Oct 30 21:39:23.284279 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:23.284259 2574 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:39:23.510025 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:23.509987 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fb540c197b86dcf7fd282ae878b96ce.slice/crio-f51fed7eefea1361370e77231a9e940aeb3c0b3eed8d48dc79deca5340db157c WatchSource:0}: Error finding container f51fed7eefea1361370e77231a9e940aeb3c0b3eed8d48dc79deca5340db157c: Status 404 returned error can't find the container with id f51fed7eefea1361370e77231a9e940aeb3c0b3eed8d48dc79deca5340db157c Oct 30 21:39:23.510511 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:23.510490 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod952d6c48a2bc0d00a5a217ec8deb4cd1.slice/crio-7e8c214acf99ded07a7499c3db8d9143ded44c04b3ade20919ebe4d194b3c31f WatchSource:0}: Error finding container 7e8c214acf99ded07a7499c3db8d9143ded44c04b3ade20919ebe4d194b3c31f: Status 404 returned error can't find the container with id 7e8c214acf99ded07a7499c3db8d9143ded44c04b3ade20919ebe4d194b3c31f Oct 30 21:39:23.515200 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:23.515185 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:39:23.665240 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:23.665214 2574 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:39:24.134936 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.134900 2574 apiserver.go:52] "Watching apiserver" Oct 30 21:39:24.153572 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.153538 2574 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 21:39:24.155341 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.155312 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-pnfw6","openshift-multus/multus-25nw6","openshift-image-registry/node-ca-m7qjt","openshift-multus/multus-additional-cni-plugins-j8vz9","kube-system/kube-apiserver-proxy-ip-10-0-137-181.us-east-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-181.us-east-2.compute.internal","openshift-multus/network-metrics-daemon-68gsj","openshift-network-diagnostics/network-check-target-9h6ck","openshift-network-operator/iptables-alerter-bzhn4","openshift-ovn-kubernetes/ovnkube-node-75z8j","kube-system/global-pull-secret-syncer-vckqz","kube-system/konnectivity-agent-jg4wr","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kf9nx","openshift-cluster-node-tuning-operator/tuned-vsgzd"] Oct 30 21:39:24.156816 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.156794 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 21:39:24.158277 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.158250 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-68gsj" Oct 30 21:39:24.158414 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:24.158353 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-68gsj" podUID="0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4" Oct 30 21:39:24.160610 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.160591 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vckqz" Oct 30 21:39:24.160713 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:24.160672 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vckqz" podUID="17127db1-cf66-4951-ba5d-eb486ee9d3d5" Oct 30 21:39:24.162982 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.162923 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-m7qjt" Oct 30 21:39:24.165809 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.165786 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-j8vz9" Oct 30 21:39:24.166566 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.166544 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4-metrics-certs\") pod \"network-metrics-daemon-68gsj\" (UID: \"0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4\") " pod="openshift-multus/network-metrics-daemon-68gsj" Oct 30 21:39:24.166672 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.166582 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbcgc\" (UniqueName: \"kubernetes.io/projected/0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4-kube-api-access-pbcgc\") pod \"network-metrics-daemon-68gsj\" (UID: \"0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4\") " pod="openshift-multus/network-metrics-daemon-68gsj" Oct 30 21:39:24.167936 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.167917 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vsgzd" Oct 30 21:39:24.171737 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.171718 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9h6ck" Oct 30 21:39:24.171830 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:24.171805 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9h6ck" podUID="df51787b-87eb-420c-aa69-3d2762bdd386" Oct 30 21:39:24.173986 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.173960 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bzhn4" Oct 30 21:39:24.178116 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.178095 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kf9nx" Oct 30 21:39:24.181165 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.181140 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-25nw6" Oct 30 21:39:24.184783 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.184764 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:39:24.187673 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.187653 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-pnfw6" Oct 30 21:39:24.190482 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.190440 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jg4wr" Oct 30 21:39:24.215121 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.215102 2574 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:39:24.233715 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.233684 2574 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 21:39:24.233823 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.233774 2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-9tfds" Oct 30 21:39:24.244822 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.244796 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-9tfds" Oct 30 21:39:24.258219 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.258195 2574 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 21:39:24.267718 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.267690 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/64995343-c3c1-4915-953a-d1e4a5252461-sys-fs\") pod \"aws-ebs-csi-driver-node-kf9nx\" (UID: \"64995343-c3c1-4915-953a-d1e4a5252461\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kf9nx" Oct 30 21:39:24.267837 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.267736 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-cni-binary-copy\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:39:24.267837 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.267775 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgrp4\" (UniqueName: \"kubernetes.io/projected/77670d00-a9cb-4607-bfb3-61ae91253cca-kube-api-access-mgrp4\") pod \"node-resolver-pnfw6\" (UID: \"77670d00-a9cb-4607-bfb3-61ae91253cca\") " pod="openshift-dns/node-resolver-pnfw6" Oct 30 21:39:24.267837 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.267805 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7d451a16-523d-4b43-95cc-de9eeffd8402-tuning-conf-dir\") pod \"multus-additional-cni-plugins-j8vz9\" (UID: \"7d451a16-523d-4b43-95cc-de9eeffd8402\") " pod="openshift-multus/multus-additional-cni-plugins-j8vz9" Oct 30 21:39:24.267837 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.267830 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/970cd7bd-9018-4f3c-976a-af47bab1460a-tmp\") pod \"tuned-vsgzd\" (UID: \"970cd7bd-9018-4f3c-976a-af47bab1460a\") " pod="openshift-cluster-node-tuning-operator/tuned-vsgzd" Oct 30 21:39:24.268007 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.267854 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-run-openvswitch\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:39:24.268007 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.267879 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-host-cni-bin\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:39:24.268007 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.267903 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-host-cni-netd\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:39:24.268007 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.267929 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/970cd7bd-9018-4f3c-976a-af47bab1460a-etc-modprobe-d\") pod \"tuned-vsgzd\" (UID: \"970cd7bd-9018-4f3c-976a-af47bab1460a\") " pod="openshift-cluster-node-tuning-operator/tuned-vsgzd" Oct 30 21:39:24.268007 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.267953 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-host-var-lib-cni-multus\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:39:24.268007 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.267976 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/64995343-c3c1-4915-953a-d1e4a5252461-device-dir\") pod \"aws-ebs-csi-driver-node-kf9nx\" (UID: \"64995343-c3c1-4915-953a-d1e4a5252461\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kf9nx" Oct 30 21:39:24.268300 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.268023 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c85rm\" (UniqueName: \"kubernetes.io/projected/64995343-c3c1-4915-953a-d1e4a5252461-kube-api-access-c85rm\") pod \"aws-ebs-csi-driver-node-kf9nx\" (UID: \"64995343-c3c1-4915-953a-d1e4a5252461\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kf9nx" Oct 30 21:39:24.268300 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.268070 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-run-systemd\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:39:24.268300 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.268128 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-etc-openvswitch\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:39:24.268300 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.268171 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:39:24.268300 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.268205 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17127db1-cf66-4951-ba5d-eb486ee9d3d5-original-pull-secret\") pod \"global-pull-secret-syncer-vckqz\" (UID: \"17127db1-cf66-4951-ba5d-eb486ee9d3d5\") " pod="kube-system/global-pull-secret-syncer-vckqz" Oct 30 21:39:24.268300 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.268242 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/970cd7bd-9018-4f3c-976a-af47bab1460a-etc-sysconfig\") pod \"tuned-vsgzd\" (UID: \"970cd7bd-9018-4f3c-976a-af47bab1460a\") " pod="openshift-cluster-node-tuning-operator/tuned-vsgzd" Oct 30 21:39:24.268300 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.268291 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbcgc\" (UniqueName: \"kubernetes.io/projected/0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4-kube-api-access-pbcgc\") pod \"network-metrics-daemon-68gsj\" (UID: \"0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4\") " pod="openshift-multus/network-metrics-daemon-68gsj" Oct 30 21:39:24.268609 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.268364 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d0a1535a-33cb-4a99-899d-aa8b3333326f-host-slash\") pod \"iptables-alerter-bzhn4\" (UID: \"d0a1535a-33cb-4a99-899d-aa8b3333326f\") " pod="openshift-network-operator/iptables-alerter-bzhn4" Oct 30 21:39:24.268609 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.268487 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2stzq\" (UniqueName: \"kubernetes.io/projected/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-kube-api-access-2stzq\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:39:24.268609 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.268539 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sl5w\" (UniqueName: \"kubernetes.io/projected/8b0ede8c-1c7c-479c-8c04-29d849fdbf43-kube-api-access-6sl5w\") pod \"node-ca-m7qjt\" (UID: \"8b0ede8c-1c7c-479c-8c04-29d849fdbf43\") " pod="openshift-image-registry/node-ca-m7qjt" Oct 30 21:39:24.268768 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.268607 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l725d\" (UniqueName: \"kubernetes.io/projected/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-kube-api-access-l725d\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:39:24.268768 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.268651 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/970cd7bd-9018-4f3c-976a-af47bab1460a-etc-sysctl-d\") pod \"tuned-vsgzd\" (UID: \"970cd7bd-9018-4f3c-976a-af47bab1460a\") " pod="openshift-cluster-node-tuning-operator/tuned-vsgzd" Oct 30 21:39:24.268768 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.268692 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/970cd7bd-9018-4f3c-976a-af47bab1460a-etc-sysctl-conf\") pod \"tuned-vsgzd\" (UID: \"970cd7bd-9018-4f3c-976a-af47bab1460a\") " pod="openshift-cluster-node-tuning-operator/tuned-vsgzd" Oct 30 21:39:24.268768 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.268749 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/64995343-c3c1-4915-953a-d1e4a5252461-etc-selinux\") pod \"aws-ebs-csi-driver-node-kf9nx\" (UID: \"64995343-c3c1-4915-953a-d1e4a5252461\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kf9nx" Oct 30 21:39:24.268939 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.268793 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4-metrics-certs\") pod \"network-metrics-daemon-68gsj\" (UID: \"0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4\") " pod="openshift-multus/network-metrics-daemon-68gsj" Oct 30 21:39:24.268939 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.268823 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-var-lib-openvswitch\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:39:24.268939 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.268852 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-host-run-ovn-kubernetes\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:39:24.268939 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:24.268898 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:39:24.268939 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.268897 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tpzc\" (UniqueName: \"kubernetes.io/projected/df51787b-87eb-420c-aa69-3d2762bdd386-kube-api-access-5tpzc\") pod \"network-check-target-9h6ck\" (UID: \"df51787b-87eb-420c-aa69-3d2762bdd386\") " pod="openshift-network-diagnostics/network-check-target-9h6ck" Oct 30 21:39:24.269141 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:24.268979 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4-metrics-certs podName:0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4 nodeName:}" failed. No retries permitted until 2026-10-30 21:39:24.768955094 +0000 UTC m=+2.976534035 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4-metrics-certs") pod "network-metrics-daemon-68gsj" (UID: "0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:39:24.269141 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.269020 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/17127db1-cf66-4951-ba5d-eb486ee9d3d5-dbus\") pod \"global-pull-secret-syncer-vckqz\" (UID: \"17127db1-cf66-4951-ba5d-eb486ee9d3d5\") " pod="kube-system/global-pull-secret-syncer-vckqz" Oct 30 21:39:24.269141 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.269077 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-systemd-units\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:39:24.269141 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.269121 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-system-cni-dir\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:39:24.269320 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.269158 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-host-run-multus-certs\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:39:24.269320 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.269193 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/27f7c44f-0287-4608-a04a-7f108a25b121-agent-certs\") pod \"konnectivity-agent-jg4wr\" (UID: \"27f7c44f-0287-4608-a04a-7f108a25b121\") " pod="kube-system/konnectivity-agent-jg4wr" Oct 30 21:39:24.269320 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.269223 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/970cd7bd-9018-4f3c-976a-af47bab1460a-etc-kubernetes\") pod \"tuned-vsgzd\" (UID: \"970cd7bd-9018-4f3c-976a-af47bab1460a\") " pod="openshift-cluster-node-tuning-operator/tuned-vsgzd" Oct 30 21:39:24.269320 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.269245 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d0a1535a-33cb-4a99-899d-aa8b3333326f-iptables-alerter-script\") pod \"iptables-alerter-bzhn4\" (UID: \"d0a1535a-33cb-4a99-899d-aa8b3333326f\") " pod="openshift-network-operator/iptables-alerter-bzhn4" Oct 30 21:39:24.269320 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.269288 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-etc-kubernetes\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:39:24.269535 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.269316 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/17127db1-cf66-4951-ba5d-eb486ee9d3d5-kubelet-config\") pod \"global-pull-secret-syncer-vckqz\" (UID: \"17127db1-cf66-4951-ba5d-eb486ee9d3d5\") " pod="kube-system/global-pull-secret-syncer-vckqz" Oct 30 21:39:24.269535 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.269377 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7d451a16-523d-4b43-95cc-de9eeffd8402-cnibin\") pod \"multus-additional-cni-plugins-j8vz9\" (UID: \"7d451a16-523d-4b43-95cc-de9eeffd8402\") " pod="openshift-multus/multus-additional-cni-plugins-j8vz9" Oct 30 21:39:24.269535 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.269430 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/970cd7bd-9018-4f3c-976a-af47bab1460a-etc-tuned\") pod \"tuned-vsgzd\" (UID: \"970cd7bd-9018-4f3c-976a-af47bab1460a\") " pod="openshift-cluster-node-tuning-operator/tuned-vsgzd" Oct 30 21:39:24.269535 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.269449 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8b0ede8c-1c7c-479c-8c04-29d849fdbf43-serviceca\") pod \"node-ca-m7qjt\" (UID: \"8b0ede8c-1c7c-479c-8c04-29d849fdbf43\") " pod="openshift-image-registry/node-ca-m7qjt" Oct 30 21:39:24.269535 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.269470 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-cnibin\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:39:24.269535 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.269510 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-multus-socket-dir-parent\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:39:24.269768 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.269557 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/970cd7bd-9018-4f3c-976a-af47bab1460a-host\") pod \"tuned-vsgzd\" (UID: \"970cd7bd-9018-4f3c-976a-af47bab1460a\") " pod="openshift-cluster-node-tuning-operator/tuned-vsgzd" Oct 30 21:39:24.269768 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.269615 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-host-kubelet\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:39:24.269768 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.269656 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8b0ede8c-1c7c-479c-8c04-29d849fdbf43-host\") pod \"node-ca-m7qjt\" (UID: \"8b0ede8c-1c7c-479c-8c04-29d849fdbf43\") " pod="openshift-image-registry/node-ca-m7qjt" Oct 30 21:39:24.269768 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.269686 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-host-run-k8s-cni-cncf-io\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:39:24.269768 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.269733 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-284h6\" (UniqueName: \"kubernetes.io/projected/7d451a16-523d-4b43-95cc-de9eeffd8402-kube-api-access-284h6\") pod \"multus-additional-cni-plugins-j8vz9\" (UID: \"7d451a16-523d-4b43-95cc-de9eeffd8402\") " pod="openshift-multus/multus-additional-cni-plugins-j8vz9" Oct 30 21:39:24.270023 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.269769 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/970cd7bd-9018-4f3c-976a-af47bab1460a-etc-systemd\") pod \"tuned-vsgzd\" (UID: \"970cd7bd-9018-4f3c-976a-af47bab1460a\") " pod="openshift-cluster-node-tuning-operator/tuned-vsgzd" Oct 30 21:39:24.270023 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.269838 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-host-slash\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:39:24.270023 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.269889 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-ovnkube-config\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:39:24.270023 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.269922 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-ovn-node-metrics-cert\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:39:24.270023 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.269949 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7d451a16-523d-4b43-95cc-de9eeffd8402-cni-binary-copy\") pod \"multus-additional-cni-plugins-j8vz9\" (UID: \"7d451a16-523d-4b43-95cc-de9eeffd8402\") " pod="openshift-multus/multus-additional-cni-plugins-j8vz9" Oct 30 21:39:24.270023 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.270000 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-multus-conf-dir\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:39:24.270256 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.270048 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/77670d00-a9cb-4607-bfb3-61ae91253cca-hosts-file\") pod \"node-resolver-pnfw6\" (UID: \"77670d00-a9cb-4607-bfb3-61ae91253cca\") " pod="openshift-dns/node-resolver-pnfw6" Oct 30 21:39:24.270256 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.270089 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-node-log\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:39:24.270256 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.270130 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj7w5\" (UniqueName: \"kubernetes.io/projected/d0a1535a-33cb-4a99-899d-aa8b3333326f-kube-api-access-sj7w5\") pod \"iptables-alerter-bzhn4\" (UID: \"d0a1535a-33cb-4a99-899d-aa8b3333326f\") " pod="openshift-network-operator/iptables-alerter-bzhn4" Oct 30 21:39:24.270256 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.270183 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-host-var-lib-cni-bin\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:39:24.270256 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.270214 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7d451a16-523d-4b43-95cc-de9eeffd8402-system-cni-dir\") pod \"multus-additional-cni-plugins-j8vz9\" (UID: \"7d451a16-523d-4b43-95cc-de9eeffd8402\") " pod="openshift-multus/multus-additional-cni-plugins-j8vz9" Oct 30 21:39:24.270256 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.270235 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7d451a16-523d-4b43-95cc-de9eeffd8402-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-j8vz9\" (UID: \"7d451a16-523d-4b43-95cc-de9eeffd8402\") " pod="openshift-multus/multus-additional-cni-plugins-j8vz9" Oct 30 21:39:24.270509 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.270316 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/64995343-c3c1-4915-953a-d1e4a5252461-socket-dir\") pod \"aws-ebs-csi-driver-node-kf9nx\" (UID: \"64995343-c3c1-4915-953a-d1e4a5252461\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kf9nx" Oct 30 21:39:24.270509 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.270346 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-host-run-netns\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:39:24.270509 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.270374 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-env-overrides\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:39:24.270509 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.270458 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-hostroot\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:39:24.270672 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.270508 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/970cd7bd-9018-4f3c-976a-af47bab1460a-run\") pod \"tuned-vsgzd\" (UID: \"970cd7bd-9018-4f3c-976a-af47bab1460a\") " pod="openshift-cluster-node-tuning-operator/tuned-vsgzd" Oct 30 21:39:24.270672 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.270541 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-log-socket\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:39:24.270672 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.270571 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/27f7c44f-0287-4608-a04a-7f108a25b121-konnectivity-ca\") pod \"konnectivity-agent-jg4wr\" (UID: \"27f7c44f-0287-4608-a04a-7f108a25b121\") " pod="kube-system/konnectivity-agent-jg4wr" Oct 30 21:39:24.270672 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.270604 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7d451a16-523d-4b43-95cc-de9eeffd8402-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-j8vz9\" (UID: \"7d451a16-523d-4b43-95cc-de9eeffd8402\") " pod="openshift-multus/multus-additional-cni-plugins-j8vz9" Oct 30 21:39:24.270672 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.270659 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-multus-cni-dir\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:39:24.270855 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.270687 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-host-run-netns\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:39:24.270855 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.270717 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-multus-daemon-config\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:39:24.270855 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.270764 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/970cd7bd-9018-4f3c-976a-af47bab1460a-sys\") pod \"tuned-vsgzd\" (UID: \"970cd7bd-9018-4f3c-976a-af47bab1460a\") " pod="openshift-cluster-node-tuning-operator/tuned-vsgzd" Oct 30 21:39:24.270855 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.270786 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-run-ovn\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:39:24.270855 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.270802 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/64995343-c3c1-4915-953a-d1e4a5252461-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kf9nx\" (UID: \"64995343-c3c1-4915-953a-d1e4a5252461\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kf9nx" Oct 30 21:39:24.270855 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.270830 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/64995343-c3c1-4915-953a-d1e4a5252461-registration-dir\") pod \"aws-ebs-csi-driver-node-kf9nx\" (UID: \"64995343-c3c1-4915-953a-d1e4a5252461\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kf9nx" Oct 30 21:39:24.271096 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.270857 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-ovnkube-script-lib\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:39:24.271096 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.270883 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-host-var-lib-kubelet\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:39:24.271096 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.270950 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/970cd7bd-9018-4f3c-976a-af47bab1460a-lib-modules\") pod \"tuned-vsgzd\" (UID: \"970cd7bd-9018-4f3c-976a-af47bab1460a\") " pod="openshift-cluster-node-tuning-operator/tuned-vsgzd" Oct 30 21:39:24.271096 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.271024 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/970cd7bd-9018-4f3c-976a-af47bab1460a-var-lib-kubelet\") pod \"tuned-vsgzd\" (UID: \"970cd7bd-9018-4f3c-976a-af47bab1460a\") " pod="openshift-cluster-node-tuning-operator/tuned-vsgzd" Oct 30 21:39:24.271096 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.271051 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77670d00-a9cb-4607-bfb3-61ae91253cca-tmp-dir\") pod \"node-resolver-pnfw6\" (UID: \"77670d00-a9cb-4607-bfb3-61ae91253cca\") " pod="openshift-dns/node-resolver-pnfw6" Oct 30 21:39:24.271096 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.271080 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2kzt\" (UniqueName: \"kubernetes.io/projected/970cd7bd-9018-4f3c-976a-af47bab1460a-kube-api-access-b2kzt\") pod \"tuned-vsgzd\" (UID: \"970cd7bd-9018-4f3c-976a-af47bab1460a\") " pod="openshift-cluster-node-tuning-operator/tuned-vsgzd" Oct 30 21:39:24.274180 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.274156 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 21:39:24.274286 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.274165 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pp8j7\"" Oct 30 21:39:24.275483 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.275450 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 21:39:24.276210 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.276190 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-6cdkd\"" Oct 30 21:39:24.276291 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.276203 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-wpx2j\"" Oct 30 21:39:24.276291 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.276259 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-h2pxp\"" Oct 30 21:39:24.276291 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.276212 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-jbnx4\"" Oct 30 21:39:24.276291 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.276285 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tzrj6\"" Oct 30 21:39:24.276523 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.276296 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-n2fsk\"" Oct 30 21:39:24.276523 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.276325 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 21:39:24.276523 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.276443 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9c426\"" Oct 30 21:39:24.276671 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.276535 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-pnbm4\"" Oct 30 21:39:24.276671 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.276619 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 21:39:24.276671 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.276625 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 21:39:24.276874 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.276705 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 21:39:24.276874 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.276701 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 21:39:24.276874 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.276735 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 21:39:24.276874 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.276766 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 21:39:24.277513 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.277418 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 21:39:24.277513 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.277478 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 21:39:24.277513 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.277487 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 21:39:24.277702 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.277532 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 21:39:24.278519 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.278377 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:39:24.278519 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.278410 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 21:39:24.278519 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.278420 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 21:39:24.278519 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.278412 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 21:39:24.278519 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.278429 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 21:39:24.278519 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.278473 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 21:39:24.278519 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.278477 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 21:39:24.278519 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.278486 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 21:39:24.278965 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.278559 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:39:24.278965 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.278651 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 21:39:24.278965 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.278715 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 21:39:24.278965 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.278790 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 21:39:24.278965 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.278877 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 21:39:24.279403 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.279372 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 21:39:24.281419 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.281393 2574 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 21:39:24.285925 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.285904 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbcgc\" (UniqueName: \"kubernetes.io/projected/0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4-kube-api-access-pbcgc\") pod \"network-metrics-daemon-68gsj\" (UID: \"0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4\") " pod="openshift-multus/network-metrics-daemon-68gsj" Oct 30 21:39:24.371529 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.371476 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-run-ovn\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:39:24.371743 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.371530 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/64995343-c3c1-4915-953a-d1e4a5252461-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kf9nx\" (UID: \"64995343-c3c1-4915-953a-d1e4a5252461\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kf9nx" Oct 30 21:39:24.371743 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.371558 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/64995343-c3c1-4915-953a-d1e4a5252461-registration-dir\") pod \"aws-ebs-csi-driver-node-kf9nx\" (UID: \"64995343-c3c1-4915-953a-d1e4a5252461\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kf9nx" Oct 30 21:39:24.371743 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.371584 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-ovnkube-script-lib\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:39:24.371743 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.371609 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-host-var-lib-kubelet\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:39:24.371743 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.371618 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/64995343-c3c1-4915-953a-d1e4a5252461-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kf9nx\" (UID: \"64995343-c3c1-4915-953a-d1e4a5252461\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kf9nx" Oct 30 21:39:24.371743 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.371633 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/970cd7bd-9018-4f3c-976a-af47bab1460a-lib-modules\") pod \"tuned-vsgzd\" (UID: \"970cd7bd-9018-4f3c-976a-af47bab1460a\") " pod="openshift-cluster-node-tuning-operator/tuned-vsgzd" Oct 30 21:39:24.371743 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.371664 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/64995343-c3c1-4915-953a-d1e4a5252461-registration-dir\") pod \"aws-ebs-csi-driver-node-kf9nx\" (UID: \"64995343-c3c1-4915-953a-d1e4a5252461\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kf9nx" Oct 30 21:39:24.371743 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.371674 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/970cd7bd-9018-4f3c-976a-af47bab1460a-var-lib-kubelet\") pod \"tuned-vsgzd\" (UID: \"970cd7bd-9018-4f3c-976a-af47bab1460a\") " pod="openshift-cluster-node-tuning-operator/tuned-vsgzd" Oct 30 21:39:24.371743 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.371722 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/970cd7bd-9018-4f3c-976a-af47bab1460a-var-lib-kubelet\") pod \"tuned-vsgzd\" (UID: \"970cd7bd-9018-4f3c-976a-af47bab1460a\") " pod="openshift-cluster-node-tuning-operator/tuned-vsgzd" Oct 30 21:39:24.372328 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.371760 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77670d00-a9cb-4607-bfb3-61ae91253cca-tmp-dir\") pod \"node-resolver-pnfw6\" (UID: \"77670d00-a9cb-4607-bfb3-61ae91253cca\") " pod="openshift-dns/node-resolver-pnfw6" Oct 30 21:39:24.372328 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.371769 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/970cd7bd-9018-4f3c-976a-af47bab1460a-lib-modules\") pod \"tuned-vsgzd\" (UID: \"970cd7bd-9018-4f3c-976a-af47bab1460a\") " pod="openshift-cluster-node-tuning-operator/tuned-vsgzd" Oct 30 21:39:24.372328 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.371821 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-host-var-lib-kubelet\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:39:24.372328 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.371863 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2kzt\" (UniqueName: \"kubernetes.io/projected/970cd7bd-9018-4f3c-976a-af47bab1460a-kube-api-access-b2kzt\") pod \"tuned-vsgzd\" (UID: \"970cd7bd-9018-4f3c-976a-af47bab1460a\") " pod="openshift-cluster-node-tuning-operator/tuned-vsgzd" Oct 30 21:39:24.372328 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.372074 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77670d00-a9cb-4607-bfb3-61ae91253cca-tmp-dir\") pod \"node-resolver-pnfw6\" (UID: \"77670d00-a9cb-4607-bfb3-61ae91253cca\") " pod="openshift-dns/node-resolver-pnfw6" Oct 30 21:39:24.372328 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.371556 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-run-ovn\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:39:24.372328 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.372112 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/64995343-c3c1-4915-953a-d1e4a5252461-sys-fs\") pod \"aws-ebs-csi-driver-node-kf9nx\" (UID: \"64995343-c3c1-4915-953a-d1e4a5252461\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kf9nx" Oct 30 21:39:24.372328 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.372143 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-cni-binary-copy\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:39:24.372328 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.372168 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgrp4\" (UniqueName: \"kubernetes.io/projected/77670d00-a9cb-4607-bfb3-61ae91253cca-kube-api-access-mgrp4\") pod \"node-resolver-pnfw6\" (UID: \"77670d00-a9cb-4607-bfb3-61ae91253cca\") " pod="openshift-dns/node-resolver-pnfw6" Oct 30 21:39:24.372328 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.372192 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7d451a16-523d-4b43-95cc-de9eeffd8402-tuning-conf-dir\") pod \"multus-additional-cni-plugins-j8vz9\" (UID: \"7d451a16-523d-4b43-95cc-de9eeffd8402\") " pod="openshift-multus/multus-additional-cni-plugins-j8vz9" Oct 30 21:39:24.372328 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.372213 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/970cd7bd-9018-4f3c-976a-af47bab1460a-tmp\") pod \"tuned-vsgzd\" (UID: \"970cd7bd-9018-4f3c-976a-af47bab1460a\") " pod="openshift-cluster-node-tuning-operator/tuned-vsgzd" Oct 30 21:39:24.372328 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.372221 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-ovnkube-script-lib\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:39:24.372328 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.372241 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-run-openvswitch\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:39:24.372328 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.372268 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-host-cni-bin\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:39:24.372328 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.372295 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-host-cni-netd\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:39:24.373080 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.372362 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/970cd7bd-9018-4f3c-976a-af47bab1460a-etc-modprobe-d\") pod \"tuned-vsgzd\" (UID: \"970cd7bd-9018-4f3c-976a-af47bab1460a\") " pod="openshift-cluster-node-tuning-operator/tuned-vsgzd" Oct 30 21:39:24.373080 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.372390 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-host-var-lib-cni-multus\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:39:24.373080 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.372416 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/64995343-c3c1-4915-953a-d1e4a5252461-device-dir\") pod \"aws-ebs-csi-driver-node-kf9nx\" (UID: \"64995343-c3c1-4915-953a-d1e4a5252461\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kf9nx" Oct 30 21:39:24.373080 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.372442 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c85rm\" (UniqueName: \"kubernetes.io/projected/64995343-c3c1-4915-953a-d1e4a5252461-kube-api-access-c85rm\") pod \"aws-ebs-csi-driver-node-kf9nx\" (UID: \"64995343-c3c1-4915-953a-d1e4a5252461\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kf9nx" Oct 30 21:39:24.373080 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.372484 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-run-systemd\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:39:24.373080 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.372510 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-etc-openvswitch\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:39:24.373080 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.372535 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:39:24.373080 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.372585 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17127db1-cf66-4951-ba5d-eb486ee9d3d5-original-pull-secret\") pod \"global-pull-secret-syncer-vckqz\" (UID: \"17127db1-cf66-4951-ba5d-eb486ee9d3d5\") " pod="kube-system/global-pull-secret-syncer-vckqz" Oct 30 21:39:24.373080 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.372610 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/970cd7bd-9018-4f3c-976a-af47bab1460a-etc-sysconfig\") pod \"tuned-vsgzd\" (UID: \"970cd7bd-9018-4f3c-976a-af47bab1460a\") " pod="openshift-cluster-node-tuning-operator/tuned-vsgzd" Oct 30 21:39:24.373080 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.372639 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d0a1535a-33cb-4a99-899d-aa8b3333326f-host-slash\") pod \"iptables-alerter-bzhn4\" (UID: \"d0a1535a-33cb-4a99-899d-aa8b3333326f\") " pod="openshift-network-operator/iptables-alerter-bzhn4" Oct 30 21:39:24.373080 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.372656 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7d451a16-523d-4b43-95cc-de9eeffd8402-tuning-conf-dir\") pod \"multus-additional-cni-plugins-j8vz9\" (UID: \"7d451a16-523d-4b43-95cc-de9eeffd8402\") " pod="openshift-multus/multus-additional-cni-plugins-j8vz9" Oct 30 21:39:24.373080 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.372665 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-cni-binary-copy\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:39:24.373080 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.372666 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2stzq\" (UniqueName: \"kubernetes.io/projected/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-kube-api-access-2stzq\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:39:24.373080 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.372704 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sl5w\" (UniqueName: \"kubernetes.io/projected/8b0ede8c-1c7c-479c-8c04-29d849fdbf43-kube-api-access-6sl5w\") pod \"node-ca-m7qjt\" (UID: \"8b0ede8c-1c7c-479c-8c04-29d849fdbf43\") " pod="openshift-image-registry/node-ca-m7qjt" Oct 30 21:39:24.373080 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.372729 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l725d\" (UniqueName: \"kubernetes.io/projected/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-kube-api-access-l725d\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:39:24.373080 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.372734 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/64995343-c3c1-4915-953a-d1e4a5252461-device-dir\") pod \"aws-ebs-csi-driver-node-kf9nx\" (UID: \"64995343-c3c1-4915-953a-d1e4a5252461\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kf9nx" Oct 30 21:39:24.373080 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.372216 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/64995343-c3c1-4915-953a-d1e4a5252461-sys-fs\") pod \"aws-ebs-csi-driver-node-kf9nx\" (UID: \"64995343-c3c1-4915-953a-d1e4a5252461\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kf9nx" Oct 30 21:39:24.373948 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.372755 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/970cd7bd-9018-4f3c-976a-af47bab1460a-etc-sysctl-d\") pod \"tuned-vsgzd\" (UID: \"970cd7bd-9018-4f3c-976a-af47bab1460a\") " pod="openshift-cluster-node-tuning-operator/tuned-vsgzd" Oct 30 21:39:24.373948 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.372797 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/970cd7bd-9018-4f3c-976a-af47bab1460a-etc-sysctl-conf\") pod \"tuned-vsgzd\" (UID: \"970cd7bd-9018-4f3c-976a-af47bab1460a\") " pod="openshift-cluster-node-tuning-operator/tuned-vsgzd" Oct 30 21:39:24.373948 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.372809 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-run-systemd\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:39:24.373948 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.372838 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/64995343-c3c1-4915-953a-d1e4a5252461-etc-selinux\") pod \"aws-ebs-csi-driver-node-kf9nx\" (UID: \"64995343-c3c1-4915-953a-d1e4a5252461\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kf9nx" Oct 30 21:39:24.373948 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.372893 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/970cd7bd-9018-4f3c-976a-af47bab1460a-etc-sysctl-d\") pod \"tuned-vsgzd\" (UID: \"970cd7bd-9018-4f3c-976a-af47bab1460a\") " pod="openshift-cluster-node-tuning-operator/tuned-vsgzd" Oct 30 21:39:24.373948 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.372903 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d0a1535a-33cb-4a99-899d-aa8b3333326f-host-slash\") pod \"iptables-alerter-bzhn4\" (UID: \"d0a1535a-33cb-4a99-899d-aa8b3333326f\") " pod="openshift-network-operator/iptables-alerter-bzhn4" Oct 30 21:39:24.373948 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.372951 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/970cd7bd-9018-4f3c-976a-af47bab1460a-etc-sysconfig\") pod \"tuned-vsgzd\" (UID: \"970cd7bd-9018-4f3c-976a-af47bab1460a\") " pod="openshift-cluster-node-tuning-operator/tuned-vsgzd" Oct 30 21:39:24.373948 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:24.373001 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:39:24.373948 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.373027 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/970cd7bd-9018-4f3c-976a-af47bab1460a-etc-modprobe-d\") pod \"tuned-vsgzd\" (UID: \"970cd7bd-9018-4f3c-976a-af47bab1460a\") " pod="openshift-cluster-node-tuning-operator/tuned-vsgzd" Oct 30 21:39:24.373948 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:24.373059 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17127db1-cf66-4951-ba5d-eb486ee9d3d5-original-pull-secret podName:17127db1-cf66-4951-ba5d-eb486ee9d3d5 nodeName:}" failed. No retries permitted until 2026-10-30 21:39:24.87304146 +0000 UTC m=+3.080620404 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/17127db1-cf66-4951-ba5d-eb486ee9d3d5-original-pull-secret") pod "global-pull-secret-syncer-vckqz" (UID: "17127db1-cf66-4951-ba5d-eb486ee9d3d5") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:39:24.373948 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.373088 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-var-lib-openvswitch\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:39:24.373948 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.373119 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-host-run-ovn-kubernetes\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:39:24.373948 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.373194 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/64995343-c3c1-4915-953a-d1e4a5252461-etc-selinux\") pod \"aws-ebs-csi-driver-node-kf9nx\" (UID: \"64995343-c3c1-4915-953a-d1e4a5252461\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kf9nx" Oct 30 21:39:24.373948 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.373210 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-run-openvswitch\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:39:24.373948 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.373211 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-host-run-ovn-kubernetes\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:39:24.373948 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.373249 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-host-cni-bin\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:39:24.373948 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.373286 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-host-cni-netd\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:39:24.375198 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.373292 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-host-var-lib-cni-multus\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:39:24.375198 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.373301 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/970cd7bd-9018-4f3c-976a-af47bab1460a-etc-sysctl-conf\") pod \"tuned-vsgzd\" (UID: \"970cd7bd-9018-4f3c-976a-af47bab1460a\") " pod="openshift-cluster-node-tuning-operator/tuned-vsgzd" Oct 30 21:39:24.375198 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.372861 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-etc-openvswitch\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:39:24.375198 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.373366 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tpzc\" (UniqueName: \"kubernetes.io/projected/df51787b-87eb-420c-aa69-3d2762bdd386-kube-api-access-5tpzc\") pod \"network-check-target-9h6ck\" (UID: \"df51787b-87eb-420c-aa69-3d2762bdd386\") " pod="openshift-network-diagnostics/network-check-target-9h6ck" Oct 30 21:39:24.375198 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.373401 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/17127db1-cf66-4951-ba5d-eb486ee9d3d5-dbus\") pod \"global-pull-secret-syncer-vckqz\" (UID: \"17127db1-cf66-4951-ba5d-eb486ee9d3d5\") " pod="kube-system/global-pull-secret-syncer-vckqz" Oct 30 21:39:24.375198 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.373433 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-systemd-units\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:39:24.375198 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.373452 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-var-lib-openvswitch\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:39:24.375198 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.373481 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-system-cni-dir\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:39:24.375198 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.373529 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-host-run-multus-certs\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:39:24.375198 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.373536 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-system-cni-dir\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:39:24.375198 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.372749 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:39:24.375198 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.373560 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/27f7c44f-0287-4608-a04a-7f108a25b121-agent-certs\") pod \"konnectivity-agent-jg4wr\" (UID: \"27f7c44f-0287-4608-a04a-7f108a25b121\") " pod="kube-system/konnectivity-agent-jg4wr" Oct 30 21:39:24.375198 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.373584 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-systemd-units\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:39:24.375198 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.373590 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/970cd7bd-9018-4f3c-976a-af47bab1460a-etc-kubernetes\") pod \"tuned-vsgzd\" (UID: \"970cd7bd-9018-4f3c-976a-af47bab1460a\") " pod="openshift-cluster-node-tuning-operator/tuned-vsgzd" Oct 30 21:39:24.375198 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.373608 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-host-run-multus-certs\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:39:24.375198 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.373619 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d0a1535a-33cb-4a99-899d-aa8b3333326f-iptables-alerter-script\") pod \"iptables-alerter-bzhn4\" (UID: \"d0a1535a-33cb-4a99-899d-aa8b3333326f\") " pod="openshift-network-operator/iptables-alerter-bzhn4" Oct 30 21:39:24.375198 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.373647 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-etc-kubernetes\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:39:24.376174 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.373666 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/970cd7bd-9018-4f3c-976a-af47bab1460a-etc-kubernetes\") pod \"tuned-vsgzd\" (UID: \"970cd7bd-9018-4f3c-976a-af47bab1460a\") " pod="openshift-cluster-node-tuning-operator/tuned-vsgzd" Oct 30 21:39:24.376174 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.373675 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/17127db1-cf66-4951-ba5d-eb486ee9d3d5-kubelet-config\") pod \"global-pull-secret-syncer-vckqz\" (UID: \"17127db1-cf66-4951-ba5d-eb486ee9d3d5\") " pod="kube-system/global-pull-secret-syncer-vckqz" Oct 30 21:39:24.376174 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.373667 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/17127db1-cf66-4951-ba5d-eb486ee9d3d5-dbus\") pod \"global-pull-secret-syncer-vckqz\" (UID: \"17127db1-cf66-4951-ba5d-eb486ee9d3d5\") " pod="kube-system/global-pull-secret-syncer-vckqz" Oct 30 21:39:24.376174 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.373705 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7d451a16-523d-4b43-95cc-de9eeffd8402-cnibin\") pod \"multus-additional-cni-plugins-j8vz9\" (UID: \"7d451a16-523d-4b43-95cc-de9eeffd8402\") " pod="openshift-multus/multus-additional-cni-plugins-j8vz9" Oct 30 21:39:24.376174 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.373733 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/970cd7bd-9018-4f3c-976a-af47bab1460a-etc-tuned\") pod \"tuned-vsgzd\" (UID: \"970cd7bd-9018-4f3c-976a-af47bab1460a\") " pod="openshift-cluster-node-tuning-operator/tuned-vsgzd" Oct 30 21:39:24.376174 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.373751 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-etc-kubernetes\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:39:24.376174 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.373757 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8b0ede8c-1c7c-479c-8c04-29d849fdbf43-serviceca\") pod \"node-ca-m7qjt\" (UID: \"8b0ede8c-1c7c-479c-8c04-29d849fdbf43\") " pod="openshift-image-registry/node-ca-m7qjt" Oct 30 21:39:24.376174 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.373784 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-cnibin\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:39:24.376174 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.373801 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7d451a16-523d-4b43-95cc-de9eeffd8402-cnibin\") pod \"multus-additional-cni-plugins-j8vz9\" (UID: \"7d451a16-523d-4b43-95cc-de9eeffd8402\") " pod="openshift-multus/multus-additional-cni-plugins-j8vz9" Oct 30 21:39:24.376174 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.373809 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-multus-socket-dir-parent\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:39:24.376174 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.373853 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/17127db1-cf66-4951-ba5d-eb486ee9d3d5-kubelet-config\") pod \"global-pull-secret-syncer-vckqz\" (UID: \"17127db1-cf66-4951-ba5d-eb486ee9d3d5\") " pod="kube-system/global-pull-secret-syncer-vckqz" Oct 30 21:39:24.376174 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.373876 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-multus-socket-dir-parent\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:39:24.376174 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.373901 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/970cd7bd-9018-4f3c-976a-af47bab1460a-host\") pod \"tuned-vsgzd\" (UID: \"970cd7bd-9018-4f3c-976a-af47bab1460a\") " pod="openshift-cluster-node-tuning-operator/tuned-vsgzd" Oct 30 21:39:24.376174 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.373923 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-host-kubelet\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:39:24.376174 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.373939 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8b0ede8c-1c7c-479c-8c04-29d849fdbf43-host\") pod \"node-ca-m7qjt\" (UID: \"8b0ede8c-1c7c-479c-8c04-29d849fdbf43\") " pod="openshift-image-registry/node-ca-m7qjt" Oct 30 21:39:24.376174 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.373954 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-host-run-k8s-cni-cncf-io\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:39:24.376174 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.373981 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-284h6\" (UniqueName: \"kubernetes.io/projected/7d451a16-523d-4b43-95cc-de9eeffd8402-kube-api-access-284h6\") pod \"multus-additional-cni-plugins-j8vz9\" (UID: \"7d451a16-523d-4b43-95cc-de9eeffd8402\") " pod="openshift-multus/multus-additional-cni-plugins-j8vz9" Oct 30 21:39:24.376174 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.374009 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/970cd7bd-9018-4f3c-976a-af47bab1460a-etc-systemd\") pod \"tuned-vsgzd\" (UID: \"970cd7bd-9018-4f3c-976a-af47bab1460a\") " pod="openshift-cluster-node-tuning-operator/tuned-vsgzd" Oct 30 21:39:24.377140 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.374039 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-host-slash\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:39:24.377140 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.374064 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-ovnkube-config\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:39:24.377140 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.374091 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-ovn-node-metrics-cert\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:39:24.377140 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.374116 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7d451a16-523d-4b43-95cc-de9eeffd8402-cni-binary-copy\") pod \"multus-additional-cni-plugins-j8vz9\" (UID: \"7d451a16-523d-4b43-95cc-de9eeffd8402\") " pod="openshift-multus/multus-additional-cni-plugins-j8vz9" Oct 30 21:39:24.377140 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.374140 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-multus-conf-dir\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:39:24.377140 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.374165 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/77670d00-a9cb-4607-bfb3-61ae91253cca-hosts-file\") pod \"node-resolver-pnfw6\" (UID: \"77670d00-a9cb-4607-bfb3-61ae91253cca\") " pod="openshift-dns/node-resolver-pnfw6" Oct 30 21:39:24.377140 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.374190 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-node-log\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:39:24.377140 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.374202 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d0a1535a-33cb-4a99-899d-aa8b3333326f-iptables-alerter-script\") pod \"iptables-alerter-bzhn4\" (UID: \"d0a1535a-33cb-4a99-899d-aa8b3333326f\") " pod="openshift-network-operator/iptables-alerter-bzhn4" Oct 30 21:39:24.377140 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.374215 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj7w5\" (UniqueName: \"kubernetes.io/projected/d0a1535a-33cb-4a99-899d-aa8b3333326f-kube-api-access-sj7w5\") pod \"iptables-alerter-bzhn4\" (UID: \"d0a1535a-33cb-4a99-899d-aa8b3333326f\") " pod="openshift-network-operator/iptables-alerter-bzhn4" Oct 30 21:39:24.377140 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.374241 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-host-var-lib-cni-bin\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:39:24.377140 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.374271 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7d451a16-523d-4b43-95cc-de9eeffd8402-system-cni-dir\") pod \"multus-additional-cni-plugins-j8vz9\" (UID: \"7d451a16-523d-4b43-95cc-de9eeffd8402\") " pod="openshift-multus/multus-additional-cni-plugins-j8vz9" Oct 30 21:39:24.377140 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.374301 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7d451a16-523d-4b43-95cc-de9eeffd8402-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-j8vz9\" (UID: \"7d451a16-523d-4b43-95cc-de9eeffd8402\") " pod="openshift-multus/multus-additional-cni-plugins-j8vz9" Oct 30 21:39:24.377140 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.374329 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/64995343-c3c1-4915-953a-d1e4a5252461-socket-dir\") pod \"aws-ebs-csi-driver-node-kf9nx\" (UID: \"64995343-c3c1-4915-953a-d1e4a5252461\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kf9nx" Oct 30 21:39:24.377140 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.374355 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-host-run-netns\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:39:24.377140 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.374380 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-env-overrides\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:39:24.377140 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.374406 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-hostroot\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:39:24.377140 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.374430 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/970cd7bd-9018-4f3c-976a-af47bab1460a-run\") pod \"tuned-vsgzd\" (UID: \"970cd7bd-9018-4f3c-976a-af47bab1460a\") " pod="openshift-cluster-node-tuning-operator/tuned-vsgzd" Oct 30 21:39:24.377939 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.374441 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-host-kubelet\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:39:24.377939 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.374475 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-log-socket\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:39:24.377939 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.374487 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8b0ede8c-1c7c-479c-8c04-29d849fdbf43-host\") pod \"node-ca-m7qjt\" (UID: \"8b0ede8c-1c7c-479c-8c04-29d849fdbf43\") " pod="openshift-image-registry/node-ca-m7qjt" Oct 30 21:39:24.377939 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.374505 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/27f7c44f-0287-4608-a04a-7f108a25b121-konnectivity-ca\") pod \"konnectivity-agent-jg4wr\" (UID: \"27f7c44f-0287-4608-a04a-7f108a25b121\") " pod="kube-system/konnectivity-agent-jg4wr" Oct 30 21:39:24.377939 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.374528 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-host-run-k8s-cni-cncf-io\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:39:24.377939 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.374534 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7d451a16-523d-4b43-95cc-de9eeffd8402-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-j8vz9\" (UID: \"7d451a16-523d-4b43-95cc-de9eeffd8402\") " pod="openshift-multus/multus-additional-cni-plugins-j8vz9" Oct 30 21:39:24.377939 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.374581 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8b0ede8c-1c7c-479c-8c04-29d849fdbf43-serviceca\") pod \"node-ca-m7qjt\" (UID: \"8b0ede8c-1c7c-479c-8c04-29d849fdbf43\") " pod="openshift-image-registry/node-ca-m7qjt" Oct 30 21:39:24.377939 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.374586 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/970cd7bd-9018-4f3c-976a-af47bab1460a-etc-systemd\") pod \"tuned-vsgzd\" (UID: \"970cd7bd-9018-4f3c-976a-af47bab1460a\") " pod="openshift-cluster-node-tuning-operator/tuned-vsgzd" Oct 30 21:39:24.377939 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.374581 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-multus-conf-dir\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:39:24.377939 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.374621 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/77670d00-a9cb-4607-bfb3-61ae91253cca-hosts-file\") pod \"node-resolver-pnfw6\" (UID: \"77670d00-a9cb-4607-bfb3-61ae91253cca\") " pod="openshift-dns/node-resolver-pnfw6" Oct 30 21:39:24.377939 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.374631 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-host-slash\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:39:24.377939 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.374661 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-node-log\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:39:24.377939 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.374889 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-host-var-lib-cni-bin\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:39:24.377939 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.374961 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7d451a16-523d-4b43-95cc-de9eeffd8402-system-cni-dir\") pod \"multus-additional-cni-plugins-j8vz9\" (UID: \"7d451a16-523d-4b43-95cc-de9eeffd8402\") " pod="openshift-multus/multus-additional-cni-plugins-j8vz9" Oct 30 21:39:24.377939 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.375107 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-cnibin\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:39:24.377939 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.374406 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/970cd7bd-9018-4f3c-976a-af47bab1460a-host\") pod \"tuned-vsgzd\" (UID: \"970cd7bd-9018-4f3c-976a-af47bab1460a\") " pod="openshift-cluster-node-tuning-operator/tuned-vsgzd" Oct 30 21:39:24.377939 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.375227 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-host-run-netns\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:39:24.377939 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.375544 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7d451a16-523d-4b43-95cc-de9eeffd8402-cni-binary-copy\") pod \"multus-additional-cni-plugins-j8vz9\" (UID: \"7d451a16-523d-4b43-95cc-de9eeffd8402\") " pod="openshift-multus/multus-additional-cni-plugins-j8vz9" Oct 30 21:39:24.378821 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.375605 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-ovnkube-config\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:39:24.378821 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.376146 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7d451a16-523d-4b43-95cc-de9eeffd8402-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-j8vz9\" (UID: \"7d451a16-523d-4b43-95cc-de9eeffd8402\") " pod="openshift-multus/multus-additional-cni-plugins-j8vz9" Oct 30 21:39:24.378821 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.376200 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-multus-cni-dir\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:39:24.378821 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.376236 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-host-run-netns\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:39:24.378821 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.376266 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-multus-daemon-config\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:39:24.378821 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.376296 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/970cd7bd-9018-4f3c-976a-af47bab1460a-sys\") pod \"tuned-vsgzd\" (UID: \"970cd7bd-9018-4f3c-976a-af47bab1460a\") " pod="openshift-cluster-node-tuning-operator/tuned-vsgzd" Oct 30 21:39:24.378821 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.376381 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/970cd7bd-9018-4f3c-976a-af47bab1460a-sys\") pod \"tuned-vsgzd\" (UID: \"970cd7bd-9018-4f3c-976a-af47bab1460a\") " pod="openshift-cluster-node-tuning-operator/tuned-vsgzd" Oct 30 21:39:24.378821 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.376436 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-multus-cni-dir\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:39:24.378821 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.376528 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-host-run-netns\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:39:24.378821 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.376617 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/970cd7bd-9018-4f3c-976a-af47bab1460a-etc-tuned\") pod \"tuned-vsgzd\" (UID: \"970cd7bd-9018-4f3c-976a-af47bab1460a\") " pod="openshift-cluster-node-tuning-operator/tuned-vsgzd" Oct 30 21:39:24.378821 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.376636 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/27f7c44f-0287-4608-a04a-7f108a25b121-konnectivity-ca\") pod \"konnectivity-agent-jg4wr\" (UID: \"27f7c44f-0287-4608-a04a-7f108a25b121\") " pod="kube-system/konnectivity-agent-jg4wr" Oct 30 21:39:24.378821 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.376695 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/970cd7bd-9018-4f3c-976a-af47bab1460a-run\") pod \"tuned-vsgzd\" (UID: \"970cd7bd-9018-4f3c-976a-af47bab1460a\") " pod="openshift-cluster-node-tuning-operator/tuned-vsgzd" Oct 30 21:39:24.378821 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.376735 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-log-socket\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:39:24.378821 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.376773 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-hostroot\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:39:24.378821 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.376950 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/970cd7bd-9018-4f3c-976a-af47bab1460a-tmp\") pod \"tuned-vsgzd\" (UID: \"970cd7bd-9018-4f3c-976a-af47bab1460a\") " pod="openshift-cluster-node-tuning-operator/tuned-vsgzd" Oct 30 21:39:24.378821 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.376992 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-multus-daemon-config\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:39:24.378821 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.377074 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-env-overrides\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:39:24.378821 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.377092 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/64995343-c3c1-4915-953a-d1e4a5252461-socket-dir\") pod \"aws-ebs-csi-driver-node-kf9nx\" (UID: \"64995343-c3c1-4915-953a-d1e4a5252461\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kf9nx" Oct 30 21:39:24.379522 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.377362 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/27f7c44f-0287-4608-a04a-7f108a25b121-agent-certs\") pod \"konnectivity-agent-jg4wr\" (UID: \"27f7c44f-0287-4608-a04a-7f108a25b121\") " pod="kube-system/konnectivity-agent-jg4wr" Oct 30 21:39:24.379522 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.377485 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7d451a16-523d-4b43-95cc-de9eeffd8402-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-j8vz9\" (UID: \"7d451a16-523d-4b43-95cc-de9eeffd8402\") " pod="openshift-multus/multus-additional-cni-plugins-j8vz9" Oct 30 21:39:24.379522 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.378078 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-ovn-node-metrics-cert\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:39:24.390719 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.390416 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgrp4\" (UniqueName: \"kubernetes.io/projected/77670d00-a9cb-4607-bfb3-61ae91253cca-kube-api-access-mgrp4\") pod \"node-resolver-pnfw6\" (UID: \"77670d00-a9cb-4607-bfb3-61ae91253cca\") " pod="openshift-dns/node-resolver-pnfw6" Oct 30 21:39:24.391711 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:24.391679 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:39:24.391711 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:24.391707 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:39:24.391865 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:24.391722 2574 projected.go:196] Error preparing data for projected volume kube-api-access-5tpzc for pod openshift-network-diagnostics/network-check-target-9h6ck: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:39:24.391865 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:24.391789 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df51787b-87eb-420c-aa69-3d2762bdd386-kube-api-access-5tpzc podName:df51787b-87eb-420c-aa69-3d2762bdd386 nodeName:}" failed. No retries permitted until 2026-10-30 21:39:24.891770284 +0000 UTC m=+3.099349245 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-5tpzc" (UniqueName: "kubernetes.io/projected/df51787b-87eb-420c-aa69-3d2762bdd386-kube-api-access-5tpzc") pod "network-check-target-9h6ck" (UID: "df51787b-87eb-420c-aa69-3d2762bdd386") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:39:24.395579 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.395553 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj7w5\" (UniqueName: \"kubernetes.io/projected/d0a1535a-33cb-4a99-899d-aa8b3333326f-kube-api-access-sj7w5\") pod \"iptables-alerter-bzhn4\" (UID: \"d0a1535a-33cb-4a99-899d-aa8b3333326f\") " pod="openshift-network-operator/iptables-alerter-bzhn4" Oct 30 21:39:24.399250 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.399225 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2kzt\" (UniqueName: \"kubernetes.io/projected/970cd7bd-9018-4f3c-976a-af47bab1460a-kube-api-access-b2kzt\") pod \"tuned-vsgzd\" (UID: \"970cd7bd-9018-4f3c-976a-af47bab1460a\") " pod="openshift-cluster-node-tuning-operator/tuned-vsgzd" Oct 30 21:39:24.399448 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.399430 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2stzq\" (UniqueName: \"kubernetes.io/projected/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-kube-api-access-2stzq\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:39:24.399544 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.399429 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-284h6\" (UniqueName: \"kubernetes.io/projected/7d451a16-523d-4b43-95cc-de9eeffd8402-kube-api-access-284h6\") pod \"multus-additional-cni-plugins-j8vz9\" (UID: \"7d451a16-523d-4b43-95cc-de9eeffd8402\") " pod="openshift-multus/multus-additional-cni-plugins-j8vz9" Oct 30 21:39:24.399832 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.399805 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l725d\" (UniqueName: \"kubernetes.io/projected/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-kube-api-access-l725d\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:39:24.399903 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.399876 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c85rm\" (UniqueName: \"kubernetes.io/projected/64995343-c3c1-4915-953a-d1e4a5252461-kube-api-access-c85rm\") pod \"aws-ebs-csi-driver-node-kf9nx\" (UID: \"64995343-c3c1-4915-953a-d1e4a5252461\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kf9nx" Oct 30 21:39:24.400843 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.400816 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sl5w\" (UniqueName: \"kubernetes.io/projected/8b0ede8c-1c7c-479c-8c04-29d849fdbf43-kube-api-access-6sl5w\") pod \"node-ca-m7qjt\" (UID: \"8b0ede8c-1c7c-479c-8c04-29d849fdbf43\") " pod="openshift-image-registry/node-ca-m7qjt" Oct 30 21:39:24.426098 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.426043 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-181.us-east-2.compute.internal" event={"ID":"952d6c48a2bc0d00a5a217ec8deb4cd1","Type":"ContainerStarted","Data":"7e8c214acf99ded07a7499c3db8d9143ded44c04b3ade20919ebe4d194b3c31f"} Oct 30 21:39:24.427483 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.427454 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-181.us-east-2.compute.internal" event={"ID":"5fb540c197b86dcf7fd282ae878b96ce","Type":"ContainerStarted","Data":"f51fed7eefea1361370e77231a9e940aeb3c0b3eed8d48dc79deca5340db157c"} Oct 30 21:39:24.480616 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.480582 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-m7qjt" Oct 30 21:39:24.496244 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:24.496215 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b0ede8c_1c7c_479c_8c04_29d849fdbf43.slice/crio-9e09f6b4dee3ab435d88c7e45da628d0b0d8618010b8b72516392ff86ab1b278 WatchSource:0}: Error finding container 9e09f6b4dee3ab435d88c7e45da628d0b0d8618010b8b72516392ff86ab1b278: Status 404 returned error can't find the container with id 9e09f6b4dee3ab435d88c7e45da628d0b0d8618010b8b72516392ff86ab1b278 Oct 30 21:39:24.509840 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.509793 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-j8vz9" Oct 30 21:39:24.512759 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.512737 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vsgzd" Oct 30 21:39:24.515196 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.515169 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kf9nx" Oct 30 21:39:24.517978 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.517951 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bzhn4" Oct 30 21:39:24.523516 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:24.523488 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d451a16_523d_4b43_95cc_de9eeffd8402.slice/crio-1a470e35815603cf8b32f1f06dfaaa23198d564e9b3d32678e4ae65748d33fb0 WatchSource:0}: Error finding container 1a470e35815603cf8b32f1f06dfaaa23198d564e9b3d32678e4ae65748d33fb0: Status 404 returned error can't find the container with id 1a470e35815603cf8b32f1f06dfaaa23198d564e9b3d32678e4ae65748d33fb0 Oct 30 21:39:24.524917 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.524896 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-25nw6" Oct 30 21:39:24.531295 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:24.531257 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod970cd7bd_9018_4f3c_976a_af47bab1460a.slice/crio-207b315446bf34ce4140521b8ecde8703e06df8c95feb5a2dc710a0e8f60e821 WatchSource:0}: Error finding container 207b315446bf34ce4140521b8ecde8703e06df8c95feb5a2dc710a0e8f60e821: Status 404 returned error can't find the container with id 207b315446bf34ce4140521b8ecde8703e06df8c95feb5a2dc710a0e8f60e821 Oct 30 21:39:24.533243 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:24.533219 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64995343_c3c1_4915_953a_d1e4a5252461.slice/crio-83ed93b8c37441e466eb5e680d93417d43d73e3a252fa206e92194540699b821 WatchSource:0}: Error finding container 83ed93b8c37441e466eb5e680d93417d43d73e3a252fa206e92194540699b821: Status 404 returned error can't find the container with id 83ed93b8c37441e466eb5e680d93417d43d73e3a252fa206e92194540699b821 Oct 30 21:39:24.537606 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:24.537569 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0a1535a_33cb_4a99_899d_aa8b3333326f.slice/crio-3b9f15580bfe7c628b1b43cb22aa394e3106ed959240bab23126401001e1cae5 WatchSource:0}: Error finding container 3b9f15580bfe7c628b1b43cb22aa394e3106ed959240bab23126401001e1cae5: Status 404 returned error can't find the container with id 3b9f15580bfe7c628b1b43cb22aa394e3106ed959240bab23126401001e1cae5 Oct 30 21:39:24.543860 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:24.543830 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ffba6fe_fae3_4f37_9070_ba3d1c39e4ab.slice/crio-8ced14ceba61a4a50c109a820429cb6e2ab0b84becfe9e0c9ffbc43e96b87e9d WatchSource:0}: Error finding container 8ced14ceba61a4a50c109a820429cb6e2ab0b84becfe9e0c9ffbc43e96b87e9d: Status 404 returned error can't find the container with id 8ced14ceba61a4a50c109a820429cb6e2ab0b84becfe9e0c9ffbc43e96b87e9d Oct 30 21:39:24.556309 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.556108 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-pnfw6" Oct 30 21:39:24.560122 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.560098 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jg4wr" Oct 30 21:39:24.561736 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.561701 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:39:24.569991 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:24.569954 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77670d00_a9cb_4607_bfb3_61ae91253cca.slice/crio-12db37d2844f143df7f3300e07da7aa980792b1668f8e8d66e432292f48e0257 WatchSource:0}: Error finding container 12db37d2844f143df7f3300e07da7aa980792b1668f8e8d66e432292f48e0257: Status 404 returned error can't find the container with id 12db37d2844f143df7f3300e07da7aa980792b1668f8e8d66e432292f48e0257 Oct 30 21:39:24.573574 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:24.573454 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27f7c44f_0287_4608_a04a_7f108a25b121.slice/crio-5b27a017bc46424bb40d7916ec70f29c76c279cbd7086e1e482b5c3c4a06f2f9 WatchSource:0}: Error finding container 5b27a017bc46424bb40d7916ec70f29c76c279cbd7086e1e482b5c3c4a06f2f9: Status 404 returned error can't find the container with id 5b27a017bc46424bb40d7916ec70f29c76c279cbd7086e1e482b5c3c4a06f2f9 Oct 30 21:39:24.576714 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:24.576691 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod451ddc8c_9938_4c6a_a210_ff9a6efd7f70.slice/crio-500e7e5992fc8ffeab24db2201da6c8c5096b9410f69f62a12141ecbab6968ac WatchSource:0}: Error finding container 500e7e5992fc8ffeab24db2201da6c8c5096b9410f69f62a12141ecbab6968ac: Status 404 returned error can't find the container with id 500e7e5992fc8ffeab24db2201da6c8c5096b9410f69f62a12141ecbab6968ac Oct 30 21:39:24.778947 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.778852 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4-metrics-certs\") pod \"network-metrics-daemon-68gsj\" (UID: \"0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4\") " pod="openshift-multus/network-metrics-daemon-68gsj" Oct 30 21:39:24.779099 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:24.779047 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:39:24.779146 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:24.779120 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4-metrics-certs podName:0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4 nodeName:}" failed. No retries permitted until 2026-10-30 21:39:25.779103163 +0000 UTC m=+3.986682105 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4-metrics-certs") pod "network-metrics-daemon-68gsj" (UID: "0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:39:24.880384 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.879793 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17127db1-cf66-4951-ba5d-eb486ee9d3d5-original-pull-secret\") pod \"global-pull-secret-syncer-vckqz\" (UID: \"17127db1-cf66-4951-ba5d-eb486ee9d3d5\") " pod="kube-system/global-pull-secret-syncer-vckqz" Oct 30 21:39:24.880384 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:24.879981 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:39:24.880384 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:24.880048 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17127db1-cf66-4951-ba5d-eb486ee9d3d5-original-pull-secret podName:17127db1-cf66-4951-ba5d-eb486ee9d3d5 nodeName:}" failed. No retries permitted until 2026-10-30 21:39:25.880028735 +0000 UTC m=+4.087607690 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/17127db1-cf66-4951-ba5d-eb486ee9d3d5-original-pull-secret") pod "global-pull-secret-syncer-vckqz" (UID: "17127db1-cf66-4951-ba5d-eb486ee9d3d5") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:39:24.980173 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:24.980140 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tpzc\" (UniqueName: \"kubernetes.io/projected/df51787b-87eb-420c-aa69-3d2762bdd386-kube-api-access-5tpzc\") pod \"network-check-target-9h6ck\" (UID: \"df51787b-87eb-420c-aa69-3d2762bdd386\") " pod="openshift-network-diagnostics/network-check-target-9h6ck" Oct 30 21:39:24.980364 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:24.980327 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:39:24.980364 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:24.980352 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:39:24.980582 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:24.980367 2574 projected.go:196] Error preparing data for projected volume kube-api-access-5tpzc for pod openshift-network-diagnostics/network-check-target-9h6ck: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:39:24.980582 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:24.980455 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df51787b-87eb-420c-aa69-3d2762bdd386-kube-api-access-5tpzc podName:df51787b-87eb-420c-aa69-3d2762bdd386 nodeName:}" failed. No retries permitted until 2026-10-30 21:39:25.980434351 +0000 UTC m=+4.188013291 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-5tpzc" (UniqueName: "kubernetes.io/projected/df51787b-87eb-420c-aa69-3d2762bdd386-kube-api-access-5tpzc") pod "network-check-target-9h6ck" (UID: "df51787b-87eb-420c-aa69-3d2762bdd386") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:39:25.248427 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:25.248272 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:34:24 +0000 UTC" deadline="2028-07-05 00:23:35.351951117 +0000 UTC" Oct 30 21:39:25.248427 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:25.248339 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14714h44m10.103617286s" Oct 30 21:39:25.420905 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:25.420026 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9h6ck" Oct 30 21:39:25.420905 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:25.420172 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9h6ck" podUID="df51787b-87eb-420c-aa69-3d2762bdd386" Oct 30 21:39:25.420905 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:25.420605 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-68gsj" Oct 30 21:39:25.420905 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:25.420721 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-68gsj" podUID="0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4" Oct 30 21:39:25.420905 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:25.420781 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vckqz" Oct 30 21:39:25.420905 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:25.420852 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vckqz" podUID="17127db1-cf66-4951-ba5d-eb486ee9d3d5" Oct 30 21:39:25.440192 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:25.440158 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jg4wr" event={"ID":"27f7c44f-0287-4608-a04a-7f108a25b121","Type":"ContainerStarted","Data":"5b27a017bc46424bb40d7916ec70f29c76c279cbd7086e1e482b5c3c4a06f2f9"} Oct 30 21:39:25.455083 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:25.455039 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-pnfw6" event={"ID":"77670d00-a9cb-4607-bfb3-61ae91253cca","Type":"ContainerStarted","Data":"12db37d2844f143df7f3300e07da7aa980792b1668f8e8d66e432292f48e0257"} Oct 30 21:39:25.466752 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:25.466708 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bzhn4" event={"ID":"d0a1535a-33cb-4a99-899d-aa8b3333326f","Type":"ContainerStarted","Data":"3b9f15580bfe7c628b1b43cb22aa394e3106ed959240bab23126401001e1cae5"} Oct 30 21:39:25.469752 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:25.469714 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vsgzd" event={"ID":"970cd7bd-9018-4f3c-976a-af47bab1460a","Type":"ContainerStarted","Data":"207b315446bf34ce4140521b8ecde8703e06df8c95feb5a2dc710a0e8f60e821"} Oct 30 21:39:25.475813 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:25.475784 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-m7qjt" event={"ID":"8b0ede8c-1c7c-479c-8c04-29d849fdbf43","Type":"ContainerStarted","Data":"9e09f6b4dee3ab435d88c7e45da628d0b0d8618010b8b72516392ff86ab1b278"} Oct 30 21:39:25.482640 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:25.482611 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" event={"ID":"451ddc8c-9938-4c6a-a210-ff9a6efd7f70","Type":"ContainerStarted","Data":"500e7e5992fc8ffeab24db2201da6c8c5096b9410f69f62a12141ecbab6968ac"} Oct 30 21:39:25.488843 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:25.488788 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-25nw6" event={"ID":"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab","Type":"ContainerStarted","Data":"8ced14ceba61a4a50c109a820429cb6e2ab0b84becfe9e0c9ffbc43e96b87e9d"} Oct 30 21:39:25.491898 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:25.491871 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kf9nx" event={"ID":"64995343-c3c1-4915-953a-d1e4a5252461","Type":"ContainerStarted","Data":"83ed93b8c37441e466eb5e680d93417d43d73e3a252fa206e92194540699b821"} Oct 30 21:39:25.498246 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:25.498218 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j8vz9" event={"ID":"7d451a16-523d-4b43-95cc-de9eeffd8402","Type":"ContainerStarted","Data":"1a470e35815603cf8b32f1f06dfaaa23198d564e9b3d32678e4ae65748d33fb0"} Oct 30 21:39:25.785508 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:25.785477 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4-metrics-certs\") pod \"network-metrics-daemon-68gsj\" (UID: \"0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4\") " pod="openshift-multus/network-metrics-daemon-68gsj" Oct 30 21:39:25.785678 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:25.785659 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:39:25.785763 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:25.785741 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4-metrics-certs podName:0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4 nodeName:}" failed. No retries permitted until 2026-10-30 21:39:27.785720529 +0000 UTC m=+5.993299469 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4-metrics-certs") pod "network-metrics-daemon-68gsj" (UID: "0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:39:25.886502 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:25.886247 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17127db1-cf66-4951-ba5d-eb486ee9d3d5-original-pull-secret\") pod \"global-pull-secret-syncer-vckqz\" (UID: \"17127db1-cf66-4951-ba5d-eb486ee9d3d5\") " pod="kube-system/global-pull-secret-syncer-vckqz" Oct 30 21:39:25.886710 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:25.886687 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:39:25.886786 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:25.886752 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17127db1-cf66-4951-ba5d-eb486ee9d3d5-original-pull-secret podName:17127db1-cf66-4951-ba5d-eb486ee9d3d5 nodeName:}" failed. No retries permitted until 2026-10-30 21:39:27.886732732 +0000 UTC m=+6.094311676 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/17127db1-cf66-4951-ba5d-eb486ee9d3d5-original-pull-secret") pod "global-pull-secret-syncer-vckqz" (UID: "17127db1-cf66-4951-ba5d-eb486ee9d3d5") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:39:25.987471 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:25.987426 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tpzc\" (UniqueName: \"kubernetes.io/projected/df51787b-87eb-420c-aa69-3d2762bdd386-kube-api-access-5tpzc\") pod \"network-check-target-9h6ck\" (UID: \"df51787b-87eb-420c-aa69-3d2762bdd386\") " pod="openshift-network-diagnostics/network-check-target-9h6ck" Oct 30 21:39:25.987654 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:25.987626 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:39:25.987654 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:25.987646 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:39:25.987765 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:25.987660 2574 projected.go:196] Error preparing data for projected volume kube-api-access-5tpzc for pod openshift-network-diagnostics/network-check-target-9h6ck: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:39:25.987765 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:25.987726 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df51787b-87eb-420c-aa69-3d2762bdd386-kube-api-access-5tpzc podName:df51787b-87eb-420c-aa69-3d2762bdd386 nodeName:}" failed. No retries permitted until 2026-10-30 21:39:27.987705617 +0000 UTC m=+6.195284573 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-5tpzc" (UniqueName: "kubernetes.io/projected/df51787b-87eb-420c-aa69-3d2762bdd386-kube-api-access-5tpzc") pod "network-check-target-9h6ck" (UID: "df51787b-87eb-420c-aa69-3d2762bdd386") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:39:26.248741 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:26.248676 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:34:24 +0000 UTC" deadline="2028-07-20 12:03:28.110874016 +0000 UTC" Oct 30 21:39:26.248741 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:26.248739 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15086h24m1.862139645s" Oct 30 21:39:26.509661 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:26.509576 2574 generic.go:356] "Generic (PLEG): container finished" podID="952d6c48a2bc0d00a5a217ec8deb4cd1" containerID="8b498db6bcfaca8f1bd378bd2c97fcb7b9d3daf34ec40e61008e8fd1b7f7ee00" exitCode=0 Oct 30 21:39:26.509880 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:26.509670 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-181.us-east-2.compute.internal" event={"ID":"952d6c48a2bc0d00a5a217ec8deb4cd1","Type":"ContainerDied","Data":"8b498db6bcfaca8f1bd378bd2c97fcb7b9d3daf34ec40e61008e8fd1b7f7ee00"} Oct 30 21:39:26.513559 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:26.513531 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-181.us-east-2.compute.internal" event={"ID":"5fb540c197b86dcf7fd282ae878b96ce","Type":"ContainerStarted","Data":"5096e5426c8feab9e02ba3d6ba73bcfda77a07fd269385fe9b0e688fa35cc80b"} Oct 30 21:39:27.421799 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:27.420950 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9h6ck" Oct 30 21:39:27.421799 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:27.421085 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9h6ck" podUID="df51787b-87eb-420c-aa69-3d2762bdd386" Oct 30 21:39:27.421799 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:27.421184 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-68gsj" Oct 30 21:39:27.421799 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:27.421277 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-68gsj" podUID="0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4" Oct 30 21:39:27.421799 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:27.421339 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vckqz" Oct 30 21:39:27.421799 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:27.421418 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vckqz" podUID="17127db1-cf66-4951-ba5d-eb486ee9d3d5" Oct 30 21:39:27.529351 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:27.529281 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-181.us-east-2.compute.internal" event={"ID":"952d6c48a2bc0d00a5a217ec8deb4cd1","Type":"ContainerStarted","Data":"3784e02ec16c2a6804995b303ca638074df0681aa619200779f593ff3d47dcd3"} Oct 30 21:39:27.549370 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:27.549294 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-181.us-east-2.compute.internal" podStartSLOduration=4.549273634 podStartE2EDuration="4.549273634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:39:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:26.552067865 +0000 UTC m=+4.759646828" watchObservedRunningTime="2026-10-30 21:39:27.549273634 +0000 UTC m=+5.756852597" Oct 30 21:39:27.802559 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:27.802484 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4-metrics-certs\") pod \"network-metrics-daemon-68gsj\" (UID: \"0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4\") " pod="openshift-multus/network-metrics-daemon-68gsj" Oct 30 21:39:27.802744 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:27.802668 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:39:27.802744 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:27.802734 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4-metrics-certs podName:0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4 nodeName:}" failed. No retries permitted until 2026-10-30 21:39:31.802715723 +0000 UTC m=+10.010294721 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4-metrics-certs") pod "network-metrics-daemon-68gsj" (UID: "0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:39:27.903188 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:27.903141 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17127db1-cf66-4951-ba5d-eb486ee9d3d5-original-pull-secret\") pod \"global-pull-secret-syncer-vckqz\" (UID: \"17127db1-cf66-4951-ba5d-eb486ee9d3d5\") " pod="kube-system/global-pull-secret-syncer-vckqz" Oct 30 21:39:27.903383 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:27.903365 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:39:27.903482 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:27.903440 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17127db1-cf66-4951-ba5d-eb486ee9d3d5-original-pull-secret podName:17127db1-cf66-4951-ba5d-eb486ee9d3d5 nodeName:}" failed. No retries permitted until 2026-10-30 21:39:31.903420797 +0000 UTC m=+10.110999751 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/17127db1-cf66-4951-ba5d-eb486ee9d3d5-original-pull-secret") pod "global-pull-secret-syncer-vckqz" (UID: "17127db1-cf66-4951-ba5d-eb486ee9d3d5") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:39:28.004244 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:28.004150 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tpzc\" (UniqueName: \"kubernetes.io/projected/df51787b-87eb-420c-aa69-3d2762bdd386-kube-api-access-5tpzc\") pod \"network-check-target-9h6ck\" (UID: \"df51787b-87eb-420c-aa69-3d2762bdd386\") " pod="openshift-network-diagnostics/network-check-target-9h6ck" Oct 30 21:39:28.004406 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:28.004331 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:39:28.004406 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:28.004355 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:39:28.004406 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:28.004368 2574 projected.go:196] Error preparing data for projected volume kube-api-access-5tpzc for pod openshift-network-diagnostics/network-check-target-9h6ck: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:39:28.004613 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:28.004451 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df51787b-87eb-420c-aa69-3d2762bdd386-kube-api-access-5tpzc podName:df51787b-87eb-420c-aa69-3d2762bdd386 nodeName:}" failed. No retries permitted until 2026-10-30 21:39:32.004414516 +0000 UTC m=+10.211993476 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-5tpzc" (UniqueName: "kubernetes.io/projected/df51787b-87eb-420c-aa69-3d2762bdd386-kube-api-access-5tpzc") pod "network-check-target-9h6ck" (UID: "df51787b-87eb-420c-aa69-3d2762bdd386") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:39:28.553209 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:28.553169 2574 generic.go:356] "Generic (PLEG): container finished" podID="7d451a16-523d-4b43-95cc-de9eeffd8402" containerID="7e67e5bbc1b09eb2355170dd2dd33ce342a445d8c026cae65701dc82a7192e31" exitCode=0 Oct 30 21:39:28.554265 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:28.554237 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j8vz9" event={"ID":"7d451a16-523d-4b43-95cc-de9eeffd8402","Type":"ContainerDied","Data":"7e67e5bbc1b09eb2355170dd2dd33ce342a445d8c026cae65701dc82a7192e31"} Oct 30 21:39:28.599094 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:28.599022 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-181.us-east-2.compute.internal" podStartSLOduration=5.599002619 podStartE2EDuration="5.599002619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:39:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:27.549709717 +0000 UTC m=+5.757288679" watchObservedRunningTime="2026-10-30 21:39:28.599002619 +0000 UTC m=+6.806581583" Oct 30 21:39:29.421118 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:29.420233 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vckqz" Oct 30 21:39:29.421118 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:29.420376 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vckqz" podUID="17127db1-cf66-4951-ba5d-eb486ee9d3d5" Oct 30 21:39:29.421118 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:29.420809 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9h6ck" Oct 30 21:39:29.421118 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:29.420919 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9h6ck" podUID="df51787b-87eb-420c-aa69-3d2762bdd386" Oct 30 21:39:29.421118 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:29.420990 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-68gsj" Oct 30 21:39:29.421118 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:29.421079 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-68gsj" podUID="0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4" Oct 30 21:39:29.564405 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:29.563652 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jg4wr" event={"ID":"27f7c44f-0287-4608-a04a-7f108a25b121","Type":"ContainerStarted","Data":"dbaa71787ace7df7cfd6c307243d34fafdd69f9991e5120c94f1955910976dc3"} Oct 30 21:39:30.572786 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:30.572750 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-m7qjt" event={"ID":"8b0ede8c-1c7c-479c-8c04-29d849fdbf43","Type":"ContainerStarted","Data":"edb925ae7a130d75cd1741bac086069f0bf301f96427a3eb7c21a5992425554c"} Oct 30 21:39:30.591113 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:30.590364 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-jg4wr" podStartSLOduration=4.406406904 podStartE2EDuration="8.590344478s" totalImagesPullingTime="4.183937574s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:29.584999671 +0000 UTC m=+7.792578611" watchObservedRunningTime="2026-10-30 21:39:30.590344478 +0000 UTC m=+8.797923439" Oct 30 21:39:31.207975 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:31.207894 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:31.368076 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:31.368036 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:31.419999 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:31.419964 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vckqz" Oct 30 21:39:31.420181 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:31.420105 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vckqz" podUID="17127db1-cf66-4951-ba5d-eb486ee9d3d5" Oct 30 21:39:31.420604 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:31.420473 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9h6ck" Oct 30 21:39:31.420604 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:31.420565 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9h6ck" podUID="df51787b-87eb-420c-aa69-3d2762bdd386" Oct 30 21:39:31.420604 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:31.420602 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-68gsj" Oct 30 21:39:31.420805 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:31.420689 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-68gsj" podUID="0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4" Oct 30 21:39:31.511431 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:31.511338 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:31.550580 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:31.550547 2574 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-jg4wr" Oct 30 21:39:31.551204 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:31.551184 2574 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-jg4wr" Oct 30 21:39:31.577405 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:31.577357 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kf9nx" event={"ID":"64995343-c3c1-4915-953a-d1e4a5252461","Type":"ContainerStarted","Data":"a4b32893e580e98d1d6ac46587366dacc247fee54670f9c3819d008a5d399cf5"} Oct 30 21:39:31.585453 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:31.585369 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-m7qjt" podStartSLOduration=4.27259484 podStartE2EDuration="9.58534945s" totalImagesPullingTime="5.31275461s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:39:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:30.591095884 +0000 UTC m=+8.798674844" watchObservedRunningTime="2026-10-30 21:39:31.58534945 +0000 UTC m=+9.792928409" Oct 30 21:39:31.665600 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:31.665566 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:31.811330 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:31.811251 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:31.835257 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:31.835224 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4-metrics-certs\") pod \"network-metrics-daemon-68gsj\" (UID: \"0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4\") " pod="openshift-multus/network-metrics-daemon-68gsj" Oct 30 21:39:31.835518 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:31.835497 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:39:31.835600 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:31.835567 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4-metrics-certs podName:0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4 nodeName:}" failed. No retries permitted until 2026-10-30 21:39:39.835550473 +0000 UTC m=+18.043129423 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4-metrics-certs") pod "network-metrics-daemon-68gsj" (UID: "0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:39:31.935918 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:31.935886 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17127db1-cf66-4951-ba5d-eb486ee9d3d5-original-pull-secret\") pod \"global-pull-secret-syncer-vckqz\" (UID: \"17127db1-cf66-4951-ba5d-eb486ee9d3d5\") " pod="kube-system/global-pull-secret-syncer-vckqz" Oct 30 21:39:31.936084 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:31.936019 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:39:31.936084 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:31.936081 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17127db1-cf66-4951-ba5d-eb486ee9d3d5-original-pull-secret podName:17127db1-cf66-4951-ba5d-eb486ee9d3d5 nodeName:}" failed. No retries permitted until 2026-10-30 21:39:39.936065562 +0000 UTC m=+18.143644513 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/17127db1-cf66-4951-ba5d-eb486ee9d3d5-original-pull-secret") pod "global-pull-secret-syncer-vckqz" (UID: "17127db1-cf66-4951-ba5d-eb486ee9d3d5") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:39:31.949264 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:31.949222 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:32.037339 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:32.036694 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tpzc\" (UniqueName: \"kubernetes.io/projected/df51787b-87eb-420c-aa69-3d2762bdd386-kube-api-access-5tpzc\") pod \"network-check-target-9h6ck\" (UID: \"df51787b-87eb-420c-aa69-3d2762bdd386\") " pod="openshift-network-diagnostics/network-check-target-9h6ck" Oct 30 21:39:32.037339 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:32.036877 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:39:32.037339 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:32.036902 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:39:32.037339 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:32.036916 2574 projected.go:196] Error preparing data for projected volume kube-api-access-5tpzc for pod openshift-network-diagnostics/network-check-target-9h6ck: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:39:32.037339 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:32.036973 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df51787b-87eb-420c-aa69-3d2762bdd386-kube-api-access-5tpzc podName:df51787b-87eb-420c-aa69-3d2762bdd386 nodeName:}" failed. No retries permitted until 2026-10-30 21:39:40.036955279 +0000 UTC m=+18.244534229 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-5tpzc" (UniqueName: "kubernetes.io/projected/df51787b-87eb-420c-aa69-3d2762bdd386-kube-api-access-5tpzc") pod "network-check-target-9h6ck" (UID: "df51787b-87eb-420c-aa69-3d2762bdd386") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:39:32.087961 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:32.087831 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:32.576529 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:32.576337 2574 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 21:39:32.583060 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:32.582221 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-25nw6" event={"ID":"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab","Type":"ContainerStarted","Data":"57275f85303013dfe2c85bca183a678ddc7ebf71af6e1e5a2d074aad4c2c905f"} Oct 30 21:39:32.592333 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:32.592191 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kf9nx" event={"ID":"64995343-c3c1-4915-953a-d1e4a5252461","Type":"ContainerStarted","Data":"a174d86cd996e1b06230c611db607894845d8a0ef54dc4ea75b093bb47bd53fb"} Oct 30 21:39:32.772350 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:32.772275 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:32.931469 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:32.931428 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:33.227326 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:33.227024 2574 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T21:39:32.576357861Z","UUID":"d4ecad30-dc71-497f-b929-7da7898775fd","Handler":null,"Name":"","Endpoint":""} Oct 30 21:39:33.230271 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:33.230248 2574 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 21:39:33.230371 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:33.230281 2574 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 21:39:33.420714 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:33.420681 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vckqz" Oct 30 21:39:33.420714 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:33.420714 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9h6ck" Oct 30 21:39:33.420939 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:33.420687 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-68gsj" Oct 30 21:39:33.420939 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:33.420820 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vckqz" podUID="17127db1-cf66-4951-ba5d-eb486ee9d3d5" Oct 30 21:39:33.421043 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:33.420973 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-68gsj" podUID="0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4" Oct 30 21:39:33.421092 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:33.421067 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9h6ck" podUID="df51787b-87eb-420c-aa69-3d2762bdd386" Oct 30 21:39:33.597491 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:33.597453 2574 generic.go:356] "Generic (PLEG): container finished" podID="7d451a16-523d-4b43-95cc-de9eeffd8402" containerID="60decec7eda6240250eed6c1ea5d5a2ce7e8ce2753f77101d17084d4056b0d73" exitCode=0 Oct 30 21:39:33.598046 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:33.597544 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j8vz9" event={"ID":"7d451a16-523d-4b43-95cc-de9eeffd8402","Type":"ContainerDied","Data":"60decec7eda6240250eed6c1ea5d5a2ce7e8ce2753f77101d17084d4056b0d73"} Oct 30 21:39:33.602041 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:33.602016 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-pnfw6" event={"ID":"77670d00-a9cb-4607-bfb3-61ae91253cca","Type":"ContainerStarted","Data":"92506137f5a24510399a37164d65bd835a891ae0f0507d0a20dd26695def69c1"} Oct 30 21:39:33.629709 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:33.629650 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-25nw6" podStartSLOduration=3.715850558 podStartE2EDuration="11.629628838s" totalImagesPullingTime="7.91377828s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:39:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:32.61834072 +0000 UTC m=+10.825919678" watchObservedRunningTime="2026-10-30 21:39:33.629628838 +0000 UTC m=+11.837207797" Oct 30 21:39:33.729612 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:33.729584 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:33.892402 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:33.892357 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:34.111150 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:34.111113 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:34.500723 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:34.500692 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:34.602095 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:34.602078 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-pnfw6_77670d00-a9cb-4607-bfb3-61ae91253cca/dns-node-resolver/0.log" Oct 30 21:39:34.606220 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:34.606196 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kf9nx" event={"ID":"64995343-c3c1-4915-953a-d1e4a5252461","Type":"ContainerStarted","Data":"921a624dd4a2bb0ecc993aa563567e7c1d1c08fda08091d4656eab6b19346807"} Oct 30 21:39:34.609524 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:34.609498 2574 generic.go:356] "Generic (PLEG): container finished" podID="7d451a16-523d-4b43-95cc-de9eeffd8402" containerID="acffc7d3523044b00355e9b09ea917ff7ee5d1d8defd3090810f8ef0e44f3e1d" exitCode=0 Oct 30 21:39:34.610074 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:34.609664 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j8vz9" event={"ID":"7d451a16-523d-4b43-95cc-de9eeffd8402","Type":"ContainerDied","Data":"acffc7d3523044b00355e9b09ea917ff7ee5d1d8defd3090810f8ef0e44f3e1d"} Oct 30 21:39:34.612066 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:34.612035 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bzhn4" event={"ID":"d0a1535a-33cb-4a99-899d-aa8b3333326f","Type":"ContainerStarted","Data":"91a8c49a5d52813db70a0123101d914e814d7888c196350cc4bd8ca97ca99bcb"} Oct 30 21:39:34.628854 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:34.628781 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-pnfw6" podStartSLOduration=4.230596101 podStartE2EDuration="12.628771026s" totalImagesPullingTime="8.398174925s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:39:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:33.647551726 +0000 UTC m=+11.855130686" watchObservedRunningTime="2026-10-30 21:39:34.628771026 +0000 UTC m=+12.836349984" Oct 30 21:39:34.665230 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:34.665185 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kf9nx" podStartSLOduration=3.902681242 podStartE2EDuration="12.665167906s" totalImagesPullingTime="8.762486664s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:39:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:34.639906092 +0000 UTC m=+12.847485051" watchObservedRunningTime="2026-10-30 21:39:34.665167906 +0000 UTC m=+12.872746865" Oct 30 21:39:34.740845 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:34.740805 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:35.082300 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:35.082228 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:35.229803 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:35.229769 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:35.399313 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:35.399286 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-m7qjt_8b0ede8c-1c7c-479c-8c04-29d849fdbf43/node-ca/0.log" Oct 30 21:39:35.419929 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:35.419905 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vckqz" Oct 30 21:39:35.420108 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:35.420068 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vckqz" podUID="17127db1-cf66-4951-ba5d-eb486ee9d3d5" Oct 30 21:39:35.420477 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:35.420452 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9h6ck" Oct 30 21:39:35.420584 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:35.420558 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9h6ck" podUID="df51787b-87eb-420c-aa69-3d2762bdd386" Oct 30 21:39:35.420657 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:35.420626 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-68gsj" Oct 30 21:39:35.420733 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:35.420713 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-68gsj" podUID="0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4" Oct 30 21:39:35.615782 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:35.615700 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vsgzd" event={"ID":"970cd7bd-9018-4f3c-976a-af47bab1460a","Type":"ContainerStarted","Data":"dce3287a32889bbfba7ccc996e9357dccd521eb6916b2740d859f5494892b82a"} Oct 30 21:39:35.637010 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:35.636964 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-vsgzd" podStartSLOduration=2.810893859 podStartE2EDuration="13.636951289s" totalImagesPullingTime="10.82605743s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:39:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:35.636500728 +0000 UTC m=+13.844079686" watchObservedRunningTime="2026-10-30 21:39:35.636951289 +0000 UTC m=+13.844530247" Oct 30 21:39:35.681899 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:35.681871 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:35.890292 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:35.890216 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:36.287775 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:36.287693 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:36.484060 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:36.483980 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:36.621362 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:36.621328 2574 generic.go:356] "Generic (PLEG): container finished" podID="7d451a16-523d-4b43-95cc-de9eeffd8402" containerID="6210176d68c45fcbf4ef9ea858cac9570fc96357d3eda865dd182425f30d23cd" exitCode=0 Oct 30 21:39:36.621991 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:36.621415 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j8vz9" event={"ID":"7d451a16-523d-4b43-95cc-de9eeffd8402","Type":"ContainerDied","Data":"6210176d68c45fcbf4ef9ea858cac9570fc96357d3eda865dd182425f30d23cd"} Oct 30 21:39:36.630283 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:36.630257 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:36.648820 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:36.648780 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-bzhn4" podStartSLOduration=5.994708756 podStartE2EDuration="14.648768226s" totalImagesPullingTime="8.65405947s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:39:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:35.655345575 +0000 UTC m=+13.862924534" watchObservedRunningTime="2026-10-30 21:39:36.648768226 +0000 UTC m=+14.856347184" Oct 30 21:39:37.090514 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:37.090422 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:37.420899 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:37.420657 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vckqz" Oct 30 21:39:37.421075 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:37.421006 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vckqz" podUID="17127db1-cf66-4951-ba5d-eb486ee9d3d5" Oct 30 21:39:37.421075 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:37.420714 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9h6ck" Oct 30 21:39:37.421185 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:37.421095 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9h6ck" podUID="df51787b-87eb-420c-aa69-3d2762bdd386" Oct 30 21:39:37.421185 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:37.420657 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-68gsj" Oct 30 21:39:37.421185 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:37.421177 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-68gsj" podUID="0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4" Oct 30 21:39:37.727258 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:37.727161 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:37.878019 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:37.877984 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:38.282984 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:38.282957 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:38.482221 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:38.482187 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:38.881299 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:38.881256 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:39.188863 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:39.188791 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:39.419946 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:39.419916 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-68gsj" Oct 30 21:39:39.419946 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:39.419936 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9h6ck" Oct 30 21:39:39.420130 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:39.419917 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vckqz" Oct 30 21:39:39.420130 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:39.420025 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-68gsj" podUID="0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4" Oct 30 21:39:39.420130 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:39.420085 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vckqz" podUID="17127db1-cf66-4951-ba5d-eb486ee9d3d5" Oct 30 21:39:39.420216 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:39.420163 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9h6ck" podUID="df51787b-87eb-420c-aa69-3d2762bdd386" Oct 30 21:39:39.630292 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:39.630144 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" event={"ID":"451ddc8c-9938-4c6a-a210-ff9a6efd7f70","Type":"ContainerStarted","Data":"7f5ad883d3be6caa53bd7e32f604481c9a2576cf5765becef1989e4ed6a9b82d"} Oct 30 21:39:39.630441 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:39.630373 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" event={"ID":"451ddc8c-9938-4c6a-a210-ff9a6efd7f70","Type":"ContainerStarted","Data":"01f158e53ceb2e36fb6f4e7190be65696bfead26bf534b385993efd87f3ece7b"} Oct 30 21:39:39.632943 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:39.632915 2574 generic.go:356] "Generic (PLEG): container finished" podID="7d451a16-523d-4b43-95cc-de9eeffd8402" containerID="b59ba665a148603ebae2d612869af6b6cb0f4d5bb3c1e5b20d3bed920d470ffa" exitCode=0 Oct 30 21:39:39.633050 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:39.632959 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j8vz9" event={"ID":"7d451a16-523d-4b43-95cc-de9eeffd8402","Type":"ContainerDied","Data":"b59ba665a148603ebae2d612869af6b6cb0f4d5bb3c1e5b20d3bed920d470ffa"} Oct 30 21:39:39.678498 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:39.678468 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:39.895641 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:39.895614 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4-metrics-certs\") pod \"network-metrics-daemon-68gsj\" (UID: \"0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4\") " pod="openshift-multus/network-metrics-daemon-68gsj" Oct 30 21:39:39.895967 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:39.895731 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:39:39.895967 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:39.895778 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4-metrics-certs podName:0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4 nodeName:}" failed. No retries permitted until 2026-10-30 21:39:55.895764631 +0000 UTC m=+34.103343568 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4-metrics-certs") pod "network-metrics-daemon-68gsj" (UID: "0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:39:39.996471 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:39.996401 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17127db1-cf66-4951-ba5d-eb486ee9d3d5-original-pull-secret\") pod \"global-pull-secret-syncer-vckqz\" (UID: \"17127db1-cf66-4951-ba5d-eb486ee9d3d5\") " pod="kube-system/global-pull-secret-syncer-vckqz" Oct 30 21:39:39.996621 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:39.996519 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:39:39.996621 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:39.996566 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17127db1-cf66-4951-ba5d-eb486ee9d3d5-original-pull-secret podName:17127db1-cf66-4951-ba5d-eb486ee9d3d5 nodeName:}" failed. No retries permitted until 2026-10-30 21:39:55.996553168 +0000 UTC m=+34.204132103 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/17127db1-cf66-4951-ba5d-eb486ee9d3d5-original-pull-secret") pod "global-pull-secret-syncer-vckqz" (UID: "17127db1-cf66-4951-ba5d-eb486ee9d3d5") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:39:40.081802 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:40.081771 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:40.097315 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:40.097289 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tpzc\" (UniqueName: \"kubernetes.io/projected/df51787b-87eb-420c-aa69-3d2762bdd386-kube-api-access-5tpzc\") pod \"network-check-target-9h6ck\" (UID: \"df51787b-87eb-420c-aa69-3d2762bdd386\") " pod="openshift-network-diagnostics/network-check-target-9h6ck" Oct 30 21:39:40.097426 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:40.097414 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:39:40.097470 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:40.097429 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:39:40.097470 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:40.097438 2574 projected.go:196] Error preparing data for projected volume kube-api-access-5tpzc for pod openshift-network-diagnostics/network-check-target-9h6ck: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:39:40.097524 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:40.097486 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df51787b-87eb-420c-aa69-3d2762bdd386-kube-api-access-5tpzc podName:df51787b-87eb-420c-aa69-3d2762bdd386 nodeName:}" failed. No retries permitted until 2026-10-30 21:39:56.097473956 +0000 UTC m=+34.305052895 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-5tpzc" (UniqueName: "kubernetes.io/projected/df51787b-87eb-420c-aa69-3d2762bdd386-kube-api-access-5tpzc") pod "network-check-target-9h6ck" (UID: "df51787b-87eb-420c-aa69-3d2762bdd386") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:39:40.220510 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:40.220480 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:40.366458 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:40.366422 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:40.506217 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:40.506190 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:40.638238 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:40.638169 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" event={"ID":"451ddc8c-9938-4c6a-a210-ff9a6efd7f70","Type":"ContainerStarted","Data":"715377c22451df3e4ca40f4f0363b64d301a89226b16d190cab305979a544b89"} Oct 30 21:39:40.638365 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:40.638251 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" event={"ID":"451ddc8c-9938-4c6a-a210-ff9a6efd7f70","Type":"ContainerStarted","Data":"165699a15c81b571402bffe54672aef74db4a2b65b470bf53597a0ecbddd09e5"} Oct 30 21:39:40.638365 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:40.638346 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" event={"ID":"451ddc8c-9938-4c6a-a210-ff9a6efd7f70","Type":"ContainerStarted","Data":"4313195900dde4e27879837cffdf5d4f60783eca90bd315bf976f45bdfbaed67"} Oct 30 21:39:40.638492 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:40.638413 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" event={"ID":"451ddc8c-9938-4c6a-a210-ff9a6efd7f70","Type":"ContainerStarted","Data":"8d316bb78fc542c39f49b535d2fc42811449fefbe5a4bb4e1f0d0d6e543e2191"} Oct 30 21:39:40.641067 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:40.641044 2574 generic.go:356] "Generic (PLEG): container finished" podID="7d451a16-523d-4b43-95cc-de9eeffd8402" containerID="0e6753e335a8c6154e619eb13b8dda7efd02f1e3d559a685c4bac597fc9ca933" exitCode=0 Oct 30 21:39:40.641158 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:40.641080 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j8vz9" event={"ID":"7d451a16-523d-4b43-95cc-de9eeffd8402","Type":"ContainerDied","Data":"0e6753e335a8c6154e619eb13b8dda7efd02f1e3d559a685c4bac597fc9ca933"} Oct 30 21:39:40.654365 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:40.654338 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:40.885193 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:40.885158 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:41.420541 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:41.420516 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-68gsj" Oct 30 21:39:41.420541 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:41.420523 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vckqz" Oct 30 21:39:41.420541 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:41.420516 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9h6ck" Oct 30 21:39:41.421070 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:41.420617 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vckqz" podUID="17127db1-cf66-4951-ba5d-eb486ee9d3d5" Oct 30 21:39:41.421070 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:41.420681 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-68gsj" podUID="0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4" Oct 30 21:39:41.421070 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:41.420736 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9h6ck" podUID="df51787b-87eb-420c-aa69-3d2762bdd386" Oct 30 21:39:41.646489 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:41.646459 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j8vz9" event={"ID":"7d451a16-523d-4b43-95cc-de9eeffd8402","Type":"ContainerStarted","Data":"76c3dd4ddd79fd6c4c9b477268db124862886d4c19108b2285daa255f769b3cb"} Oct 30 21:39:41.673823 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:41.673748 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-j8vz9" podStartSLOduration=8.263639401 podStartE2EDuration="19.673736249s" totalImagesPullingTime="11.410096848s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:39:22 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:41.671931117 +0000 UTC m=+19.879510076" watchObservedRunningTime="2026-10-30 21:39:41.673736249 +0000 UTC m=+19.881315203" Oct 30 21:39:41.680597 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:41.680572 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:42.101468 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:42.101437 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:42.281981 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:42.281907 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:42.476808 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:42.476776 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:42.651772 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:42.651738 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" event={"ID":"451ddc8c-9938-4c6a-a210-ff9a6efd7f70","Type":"ContainerStarted","Data":"a3331776406a1e661bf2132096a93efb30d2f7bef36d3d7b091a0e1e18ba0faf"} Oct 30 21:39:42.887072 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:42.887042 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:42.973242 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:42.973188 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-jg4wr" Oct 30 21:39:42.973362 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:42.973323 2574 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:39:42.973724 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:42.973709 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-jg4wr" Oct 30 21:39:43.420488 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:43.420455 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9h6ck" Oct 30 21:39:43.420675 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:43.420492 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-68gsj" Oct 30 21:39:43.420675 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:43.420455 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vckqz" Oct 30 21:39:43.420675 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:43.420582 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9h6ck" podUID="df51787b-87eb-420c-aa69-3d2762bdd386" Oct 30 21:39:43.420827 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:43.420690 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vckqz" podUID="17127db1-cf66-4951-ba5d-eb486ee9d3d5" Oct 30 21:39:43.420827 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:43.420753 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-68gsj" podUID="0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4" Oct 30 21:39:43.676907 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:43.676828 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:43.878134 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:43.878098 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:44.076982 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:44.076952 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:44.232077 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:44.231887 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:44.376474 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:44.376444 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:44.659743 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:44.659669 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" event={"ID":"451ddc8c-9938-4c6a-a210-ff9a6efd7f70","Type":"ContainerStarted","Data":"ca97a0b4a43d8436c8349cc8be225fe63decf86355ee65f6b585e030112a8cf8"} Oct 30 21:39:44.659945 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:44.659923 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:39:44.659993 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:44.659984 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:39:44.677692 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:44.677671 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:39:44.678150 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:44.677762 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:39:44.697078 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:44.697041 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" podStartSLOduration=7.806324603 podStartE2EDuration="22.697030199s" totalImagesPullingTime="14.890705596s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:39:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:44.69623644 +0000 UTC m=+22.903815399" watchObservedRunningTime="2026-10-30 21:39:44.697030199 +0000 UTC m=+22.904609157" Oct 30 21:39:45.032621 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:45.032522 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:45.178948 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:45.178916 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:45.402820 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:45.402792 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:45.420927 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:45.420906 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vckqz" Oct 30 21:39:45.421075 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:45.420934 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9h6ck" Oct 30 21:39:45.421075 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:45.421003 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-68gsj" Oct 30 21:39:45.421075 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:45.421022 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vckqz" podUID="17127db1-cf66-4951-ba5d-eb486ee9d3d5" Oct 30 21:39:45.421237 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:45.421088 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-68gsj" podUID="0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4" Oct 30 21:39:45.421237 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:45.421151 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9h6ck" podUID="df51787b-87eb-420c-aa69-3d2762bdd386" Oct 30 21:39:45.534291 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:45.534260 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:45.662274 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:45.662208 2574 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:39:46.663848 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:46.663815 2574 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:39:47.420913 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:47.420883 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vckqz" Oct 30 21:39:47.421081 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:47.420969 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9h6ck" Oct 30 21:39:47.421081 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:47.420975 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vckqz" podUID="17127db1-cf66-4951-ba5d-eb486ee9d3d5" Oct 30 21:39:47.421081 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:47.421056 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9h6ck" podUID="df51787b-87eb-420c-aa69-3d2762bdd386" Oct 30 21:39:47.421081 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:47.421062 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-68gsj" Oct 30 21:39:47.421213 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:47.421124 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-68gsj" podUID="0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4" Oct 30 21:39:48.854470 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:48.854438 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:48.993755 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:48.993725 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:49.135151 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:49.135069 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:49.420566 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:49.420481 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-68gsj" Oct 30 21:39:49.420566 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:49.420514 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9h6ck" Oct 30 21:39:49.420790 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:49.420491 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vckqz" Oct 30 21:39:49.420790 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:49.420605 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-68gsj" podUID="0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4" Oct 30 21:39:49.420790 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:49.420680 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9h6ck" podUID="df51787b-87eb-420c-aa69-3d2762bdd386" Oct 30 21:39:49.420790 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:49.420733 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vckqz" podUID="17127db1-cf66-4951-ba5d-eb486ee9d3d5" Oct 30 21:39:49.989876 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:49.989844 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:50.131592 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:50.131558 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:50.281253 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:50.281175 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:50.835400 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:50.835355 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:51.017345 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:51.017313 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:51.203074 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:51.203033 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:51.347167 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:51.347130 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:51.420897 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:51.420866 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-68gsj" Oct 30 21:39:51.421062 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:51.420904 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9h6ck" Oct 30 21:39:51.421062 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:51.420998 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-68gsj" podUID="0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4" Oct 30 21:39:51.421062 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:51.421037 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9h6ck" podUID="df51787b-87eb-420c-aa69-3d2762bdd386" Oct 30 21:39:51.421225 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:51.421075 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vckqz" Oct 30 21:39:51.421225 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:51.421147 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vckqz" podUID="17127db1-cf66-4951-ba5d-eb486ee9d3d5" Oct 30 21:39:51.912689 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:51.912660 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vckqz"] Oct 30 21:39:51.912857 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:51.912783 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vckqz" Oct 30 21:39:51.912922 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:51.912894 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vckqz" podUID="17127db1-cf66-4951-ba5d-eb486ee9d3d5" Oct 30 21:39:51.920980 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:51.920950 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9h6ck"] Oct 30 21:39:51.921186 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:51.921171 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-68gsj"] Oct 30 21:39:51.921361 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:51.921339 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-68gsj" Oct 30 21:39:51.921515 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:51.921492 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-68gsj" podUID="0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4" Oct 30 21:39:51.921573 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:51.921556 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9h6ck" Oct 30 21:39:51.921648 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:51.921628 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9h6ck" podUID="df51787b-87eb-420c-aa69-3d2762bdd386" Oct 30 21:39:52.660789 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:52.660754 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:52.718397 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:52.718352 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:39:52.718706 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:52.718672 2574 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:39:52.729843 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:52.729810 2574 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" podUID="451ddc8c-9938-4c6a-a210-ff9a6efd7f70" containerName="ovnkube-controller" probeResult="failure" output="" Oct 30 21:39:52.738258 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:52.738228 2574 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" podUID="451ddc8c-9938-4c6a-a210-ff9a6efd7f70" containerName="ovnkube-controller" probeResult="failure" output="" Oct 30 21:39:52.803343 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:52.803320 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:52.950895 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:52.950831 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:53.420520 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:53.420491 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vckqz" Oct 30 21:39:53.420667 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:53.420541 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9h6ck" Oct 30 21:39:53.420667 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:53.420613 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9h6ck" podUID="df51787b-87eb-420c-aa69-3d2762bdd386" Oct 30 21:39:53.420736 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:53.420707 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-68gsj" Oct 30 21:39:53.420823 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:53.420807 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-68gsj" podUID="0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4" Oct 30 21:39:53.420901 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:53.420883 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vckqz" podUID="17127db1-cf66-4951-ba5d-eb486ee9d3d5" Oct 30 21:39:53.855719 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:53.855538 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:54.069713 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:54.069680 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:54.259904 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:54.259863 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:54.405052 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:54.404971 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:55.419877 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:55.419846 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vckqz" Oct 30 21:39:55.420443 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:55.419947 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9h6ck" Oct 30 21:39:55.420443 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:55.419960 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vckqz" podUID="17127db1-cf66-4951-ba5d-eb486ee9d3d5" Oct 30 21:39:55.420443 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:55.420029 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-68gsj" Oct 30 21:39:55.420443 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:55.420030 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9h6ck" podUID="df51787b-87eb-420c-aa69-3d2762bdd386" Oct 30 21:39:55.420443 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:55.420095 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-68gsj" podUID="0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4" Oct 30 21:39:55.905629 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:55.905596 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4-metrics-certs\") pod \"network-metrics-daemon-68gsj\" (UID: \"0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4\") " pod="openshift-multus/network-metrics-daemon-68gsj" Oct 30 21:39:55.905762 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:55.905741 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:39:55.905825 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:55.905815 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4-metrics-certs podName:0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4 nodeName:}" failed. No retries permitted until 2026-10-30 21:40:27.905799042 +0000 UTC m=+66.113377995 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4-metrics-certs") pod "network-metrics-daemon-68gsj" (UID: "0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:39:56.006779 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:56.006752 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17127db1-cf66-4951-ba5d-eb486ee9d3d5-original-pull-secret\") pod \"global-pull-secret-syncer-vckqz\" (UID: \"17127db1-cf66-4951-ba5d-eb486ee9d3d5\") " pod="kube-system/global-pull-secret-syncer-vckqz" Oct 30 21:39:56.006918 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:56.006893 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:39:56.006958 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:56.006951 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17127db1-cf66-4951-ba5d-eb486ee9d3d5-original-pull-secret podName:17127db1-cf66-4951-ba5d-eb486ee9d3d5 nodeName:}" failed. No retries permitted until 2026-10-30 21:40:28.006936404 +0000 UTC m=+66.214515339 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/17127db1-cf66-4951-ba5d-eb486ee9d3d5-original-pull-secret") pod "global-pull-secret-syncer-vckqz" (UID: "17127db1-cf66-4951-ba5d-eb486ee9d3d5") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:39:56.107774 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:56.107748 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tpzc\" (UniqueName: \"kubernetes.io/projected/df51787b-87eb-420c-aa69-3d2762bdd386-kube-api-access-5tpzc\") pod \"network-check-target-9h6ck\" (UID: \"df51787b-87eb-420c-aa69-3d2762bdd386\") " pod="openshift-network-diagnostics/network-check-target-9h6ck" Oct 30 21:39:56.107954 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:56.107879 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:39:56.107954 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:56.107893 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:39:56.107954 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:56.107902 2574 projected.go:196] Error preparing data for projected volume kube-api-access-5tpzc for pod openshift-network-diagnostics/network-check-target-9h6ck: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:39:56.107954 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:39:56.107946 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df51787b-87eb-420c-aa69-3d2762bdd386-kube-api-access-5tpzc podName:df51787b-87eb-420c-aa69-3d2762bdd386 nodeName:}" failed. No retries permitted until 2026-10-30 21:40:28.107934582 +0000 UTC m=+66.315513523 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-5tpzc" (UniqueName: "kubernetes.io/projected/df51787b-87eb-420c-aa69-3d2762bdd386-kube-api-access-5tpzc") pod "network-check-target-9h6ck" (UID: "df51787b-87eb-420c-aa69-3d2762bdd386") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:39:56.123634 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:56.123609 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-181.us-east-2.compute.internal" event="NodeReady" Oct 30 21:39:56.123728 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:56.123714 2574 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 21:39:56.175601 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:56.175575 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-v9jlm"] Oct 30 21:39:56.213913 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:56.213892 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-tj676"] Oct 30 21:39:56.214024 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:56.214010 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-v9jlm" Oct 30 21:39:56.218034 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:56.218014 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 21:39:56.218492 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:56.218465 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-v9jlm"] Oct 30 21:39:56.218492 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:56.218512 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tj676"] Oct 30 21:39:56.218686 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:56.218531 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-bhdkg"] Oct 30 21:39:56.218686 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:56.218534 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-glzqf\"" Oct 30 21:39:56.218686 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:56.218551 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 21:39:56.218686 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:56.218676 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tj676" Oct 30 21:39:56.222311 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:56.221689 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bhdkg"] Oct 30 21:39:56.222311 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:56.221782 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bhdkg" Oct 30 21:39:56.223317 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:56.223299 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 21:39:56.223706 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:56.223614 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 21:39:56.223706 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:56.223676 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-csxns\"" Oct 30 21:39:56.223706 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:56.223677 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 21:39:56.225503 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:56.225484 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 21:39:56.225583 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:56.225525 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-45g4h\"" Oct 30 21:39:56.226192 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:56.226177 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 21:39:56.226373 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:56.226356 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 21:39:56.227323 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:56.227306 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 21:39:56.309530 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:56.309501 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e2c4958f-c7a5-4b6a-9b29-c7e2cd7d2d41-config-volume\") pod \"dns-default-v9jlm\" (UID: \"e2c4958f-c7a5-4b6a-9b29-c7e2cd7d2d41\") " pod="openshift-dns/dns-default-v9jlm" Oct 30 21:39:56.309674 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:56.309542 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7x77\" (UniqueName: \"kubernetes.io/projected/3018f90d-07fd-4d10-80a3-1f14248e638c-kube-api-access-k7x77\") pod \"ingress-canary-tj676\" (UID: \"3018f90d-07fd-4d10-80a3-1f14248e638c\") " pod="openshift-ingress-canary/ingress-canary-tj676" Oct 30 21:39:56.309674 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:56.309559 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b8ed5eda-cbec-4adf-93b1-0ee0f1203f30-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bhdkg\" (UID: \"b8ed5eda-cbec-4adf-93b1-0ee0f1203f30\") " pod="openshift-insights/insights-runtime-extractor-bhdkg" Oct 30 21:39:56.309674 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:56.309575 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b8ed5eda-cbec-4adf-93b1-0ee0f1203f30-crio-socket\") pod \"insights-runtime-extractor-bhdkg\" (UID: \"b8ed5eda-cbec-4adf-93b1-0ee0f1203f30\") " pod="openshift-insights/insights-runtime-extractor-bhdkg" Oct 30 21:39:56.309805 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:56.309688 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3018f90d-07fd-4d10-80a3-1f14248e638c-cert\") pod \"ingress-canary-tj676\" (UID: \"3018f90d-07fd-4d10-80a3-1f14248e638c\") " pod="openshift-ingress-canary/ingress-canary-tj676" Oct 30 21:39:56.309805 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:56.309744 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2c4958f-c7a5-4b6a-9b29-c7e2cd7d2d41-tmp-dir\") pod \"dns-default-v9jlm\" (UID: \"e2c4958f-c7a5-4b6a-9b29-c7e2cd7d2d41\") " pod="openshift-dns/dns-default-v9jlm" Oct 30 21:39:56.309805 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:56.309767 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlvnp\" (UniqueName: \"kubernetes.io/projected/e2c4958f-c7a5-4b6a-9b29-c7e2cd7d2d41-kube-api-access-zlvnp\") pod \"dns-default-v9jlm\" (UID: \"e2c4958f-c7a5-4b6a-9b29-c7e2cd7d2d41\") " pod="openshift-dns/dns-default-v9jlm" Oct 30 21:39:56.309961 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:56.309863 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2c4958f-c7a5-4b6a-9b29-c7e2cd7d2d41-metrics-tls\") pod \"dns-default-v9jlm\" (UID: \"e2c4958f-c7a5-4b6a-9b29-c7e2cd7d2d41\") " pod="openshift-dns/dns-default-v9jlm" Oct 30 21:39:56.309961 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:56.309897 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b8ed5eda-cbec-4adf-93b1-0ee0f1203f30-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bhdkg\" (UID: \"b8ed5eda-cbec-4adf-93b1-0ee0f1203f30\") " pod="openshift-insights/insights-runtime-extractor-bhdkg" Oct 30 21:39:56.309961 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:56.309931 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm68d\" (UniqueName: \"kubernetes.io/projected/b8ed5eda-cbec-4adf-93b1-0ee0f1203f30-kube-api-access-gm68d\") pod \"insights-runtime-extractor-bhdkg\" (UID: \"b8ed5eda-cbec-4adf-93b1-0ee0f1203f30\") " pod="openshift-insights/insights-runtime-extractor-bhdkg" Oct 30 21:39:56.310061 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:56.309968 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b8ed5eda-cbec-4adf-93b1-0ee0f1203f30-data-volume\") pod \"insights-runtime-extractor-bhdkg\" (UID: \"b8ed5eda-cbec-4adf-93b1-0ee0f1203f30\") " pod="openshift-insights/insights-runtime-extractor-bhdkg" Oct 30 21:39:56.410821 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:56.410758 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2c4958f-c7a5-4b6a-9b29-c7e2cd7d2d41-tmp-dir\") pod \"dns-default-v9jlm\" (UID: \"e2c4958f-c7a5-4b6a-9b29-c7e2cd7d2d41\") " pod="openshift-dns/dns-default-v9jlm" Oct 30 21:39:56.410821 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:56.410784 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlvnp\" (UniqueName: \"kubernetes.io/projected/e2c4958f-c7a5-4b6a-9b29-c7e2cd7d2d41-kube-api-access-zlvnp\") pod \"dns-default-v9jlm\" (UID: \"e2c4958f-c7a5-4b6a-9b29-c7e2cd7d2d41\") " pod="openshift-dns/dns-default-v9jlm" Oct 30 21:39:56.410821 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:56.410806 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2c4958f-c7a5-4b6a-9b29-c7e2cd7d2d41-metrics-tls\") pod \"dns-default-v9jlm\" (UID: \"e2c4958f-c7a5-4b6a-9b29-c7e2cd7d2d41\") " pod="openshift-dns/dns-default-v9jlm" Oct 30 21:39:56.410821 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:56.410823 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b8ed5eda-cbec-4adf-93b1-0ee0f1203f30-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bhdkg\" (UID: \"b8ed5eda-cbec-4adf-93b1-0ee0f1203f30\") " pod="openshift-insights/insights-runtime-extractor-bhdkg" Oct 30 21:39:56.411090 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:56.410839 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm68d\" (UniqueName: \"kubernetes.io/projected/b8ed5eda-cbec-4adf-93b1-0ee0f1203f30-kube-api-access-gm68d\") pod \"insights-runtime-extractor-bhdkg\" (UID: \"b8ed5eda-cbec-4adf-93b1-0ee0f1203f30\") " pod="openshift-insights/insights-runtime-extractor-bhdkg" Oct 30 21:39:56.411090 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:56.410857 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b8ed5eda-cbec-4adf-93b1-0ee0f1203f30-data-volume\") pod \"insights-runtime-extractor-bhdkg\" (UID: \"b8ed5eda-cbec-4adf-93b1-0ee0f1203f30\") " pod="openshift-insights/insights-runtime-extractor-bhdkg" Oct 30 21:39:56.411090 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:56.410897 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e2c4958f-c7a5-4b6a-9b29-c7e2cd7d2d41-config-volume\") pod \"dns-default-v9jlm\" (UID: \"e2c4958f-c7a5-4b6a-9b29-c7e2cd7d2d41\") " pod="openshift-dns/dns-default-v9jlm" Oct 30 21:39:56.411090 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:56.410940 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7x77\" (UniqueName: \"kubernetes.io/projected/3018f90d-07fd-4d10-80a3-1f14248e638c-kube-api-access-k7x77\") pod \"ingress-canary-tj676\" (UID: \"3018f90d-07fd-4d10-80a3-1f14248e638c\") " pod="openshift-ingress-canary/ingress-canary-tj676" Oct 30 21:39:56.411090 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:56.410969 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b8ed5eda-cbec-4adf-93b1-0ee0f1203f30-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bhdkg\" (UID: \"b8ed5eda-cbec-4adf-93b1-0ee0f1203f30\") " pod="openshift-insights/insights-runtime-extractor-bhdkg" Oct 30 21:39:56.411090 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:56.410997 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b8ed5eda-cbec-4adf-93b1-0ee0f1203f30-crio-socket\") pod \"insights-runtime-extractor-bhdkg\" (UID: \"b8ed5eda-cbec-4adf-93b1-0ee0f1203f30\") " pod="openshift-insights/insights-runtime-extractor-bhdkg" Oct 30 21:39:56.411090 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:56.411025 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3018f90d-07fd-4d10-80a3-1f14248e638c-cert\") pod \"ingress-canary-tj676\" (UID: \"3018f90d-07fd-4d10-80a3-1f14248e638c\") " pod="openshift-ingress-canary/ingress-canary-tj676" Oct 30 21:39:56.411372 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:56.411235 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2c4958f-c7a5-4b6a-9b29-c7e2cd7d2d41-tmp-dir\") pod \"dns-default-v9jlm\" (UID: \"e2c4958f-c7a5-4b6a-9b29-c7e2cd7d2d41\") " pod="openshift-dns/dns-default-v9jlm" Oct 30 21:39:56.411372 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:56.411234 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b8ed5eda-cbec-4adf-93b1-0ee0f1203f30-data-volume\") pod \"insights-runtime-extractor-bhdkg\" (UID: \"b8ed5eda-cbec-4adf-93b1-0ee0f1203f30\") " pod="openshift-insights/insights-runtime-extractor-bhdkg" Oct 30 21:39:56.411531 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:56.411418 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b8ed5eda-cbec-4adf-93b1-0ee0f1203f30-crio-socket\") pod \"insights-runtime-extractor-bhdkg\" (UID: \"b8ed5eda-cbec-4adf-93b1-0ee0f1203f30\") " pod="openshift-insights/insights-runtime-extractor-bhdkg" Oct 30 21:39:56.411683 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:56.411658 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e2c4958f-c7a5-4b6a-9b29-c7e2cd7d2d41-config-volume\") pod \"dns-default-v9jlm\" (UID: \"e2c4958f-c7a5-4b6a-9b29-c7e2cd7d2d41\") " pod="openshift-dns/dns-default-v9jlm" Oct 30 21:39:56.411793 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:56.411683 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b8ed5eda-cbec-4adf-93b1-0ee0f1203f30-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bhdkg\" (UID: \"b8ed5eda-cbec-4adf-93b1-0ee0f1203f30\") " pod="openshift-insights/insights-runtime-extractor-bhdkg" Oct 30 21:39:56.414923 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:56.414896 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b8ed5eda-cbec-4adf-93b1-0ee0f1203f30-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bhdkg\" (UID: \"b8ed5eda-cbec-4adf-93b1-0ee0f1203f30\") " pod="openshift-insights/insights-runtime-extractor-bhdkg" Oct 30 21:39:56.415005 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:56.414925 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2c4958f-c7a5-4b6a-9b29-c7e2cd7d2d41-metrics-tls\") pod \"dns-default-v9jlm\" (UID: \"e2c4958f-c7a5-4b6a-9b29-c7e2cd7d2d41\") " pod="openshift-dns/dns-default-v9jlm" Oct 30 21:39:56.415005 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:56.414939 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3018f90d-07fd-4d10-80a3-1f14248e638c-cert\") pod \"ingress-canary-tj676\" (UID: \"3018f90d-07fd-4d10-80a3-1f14248e638c\") " pod="openshift-ingress-canary/ingress-canary-tj676" Oct 30 21:39:56.423851 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:56.423826 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlvnp\" (UniqueName: \"kubernetes.io/projected/e2c4958f-c7a5-4b6a-9b29-c7e2cd7d2d41-kube-api-access-zlvnp\") pod \"dns-default-v9jlm\" (UID: \"e2c4958f-c7a5-4b6a-9b29-c7e2cd7d2d41\") " pod="openshift-dns/dns-default-v9jlm" Oct 30 21:39:56.424208 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:56.423986 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm68d\" (UniqueName: \"kubernetes.io/projected/b8ed5eda-cbec-4adf-93b1-0ee0f1203f30-kube-api-access-gm68d\") pod \"insights-runtime-extractor-bhdkg\" (UID: \"b8ed5eda-cbec-4adf-93b1-0ee0f1203f30\") " pod="openshift-insights/insights-runtime-extractor-bhdkg" Oct 30 21:39:56.424248 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:56.424235 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7x77\" (UniqueName: \"kubernetes.io/projected/3018f90d-07fd-4d10-80a3-1f14248e638c-kube-api-access-k7x77\") pod \"ingress-canary-tj676\" (UID: \"3018f90d-07fd-4d10-80a3-1f14248e638c\") " pod="openshift-ingress-canary/ingress-canary-tj676" Oct 30 21:39:56.552839 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:56.552807 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-v9jlm" Oct 30 21:39:56.569080 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:56.569059 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tj676" Oct 30 21:39:56.580223 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:56.580205 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bhdkg" Oct 30 21:39:56.908563 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:56.908528 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tj676"] Oct 30 21:39:56.911830 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:56.911791 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-v9jlm"] Oct 30 21:39:56.912210 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:56.912181 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3018f90d_07fd_4d10_80a3_1f14248e638c.slice/crio-c1d2a886d7be5a91991de6e7e298cf3d83bd72911647599177df5163ec75b12d WatchSource:0}: Error finding container c1d2a886d7be5a91991de6e7e298cf3d83bd72911647599177df5163ec75b12d: Status 404 returned error can't find the container with id c1d2a886d7be5a91991de6e7e298cf3d83bd72911647599177df5163ec75b12d Oct 30 21:39:56.912552 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:56.912534 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bhdkg"] Oct 30 21:39:56.915811 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:56.915792 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8ed5eda_cbec_4adf_93b1_0ee0f1203f30.slice/crio-eecd9db481580b49d59a74dcbc9a329ce9448bd64ee9ea4929683f64daf18753 WatchSource:0}: Error finding container eecd9db481580b49d59a74dcbc9a329ce9448bd64ee9ea4929683f64daf18753: Status 404 returned error can't find the container with id eecd9db481580b49d59a74dcbc9a329ce9448bd64ee9ea4929683f64daf18753 Oct 30 21:39:56.916884 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:56.916864 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2c4958f_c7a5_4b6a_9b29_c7e2cd7d2d41.slice/crio-b046cd77c79cc8e11522200053c9c9c9312de2b9196a0fcbf296104cfeeefcc5 WatchSource:0}: Error finding container b046cd77c79cc8e11522200053c9c9c9312de2b9196a0fcbf296104cfeeefcc5: Status 404 returned error can't find the container with id b046cd77c79cc8e11522200053c9c9c9312de2b9196a0fcbf296104cfeeefcc5 Oct 30 21:39:57.420978 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:57.420807 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9h6ck" Oct 30 21:39:57.421145 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:57.420834 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vckqz" Oct 30 21:39:57.421226 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:57.420841 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-68gsj" Oct 30 21:39:57.425453 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:57.425424 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 21:39:57.426108 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:57.425582 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 21:39:57.427062 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:57.427040 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-scjcc\"" Oct 30 21:39:57.427192 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:57.427161 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-ggbv4\"" Oct 30 21:39:57.427192 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:57.427176 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 21:39:57.427323 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:57.427195 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 21:39:57.427323 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:57.427197 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4mh2p\"" Oct 30 21:39:57.476905 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:57.476884 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-pnfw6_77670d00-a9cb-4607-bfb3-61ae91253cca/dns-node-resolver/0.log" Oct 30 21:39:57.691913 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:57.691829 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tj676" event={"ID":"3018f90d-07fd-4d10-80a3-1f14248e638c","Type":"ContainerStarted","Data":"c1d2a886d7be5a91991de6e7e298cf3d83bd72911647599177df5163ec75b12d"} Oct 30 21:39:57.693640 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:57.693609 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-v9jlm" event={"ID":"e2c4958f-c7a5-4b6a-9b29-c7e2cd7d2d41","Type":"ContainerStarted","Data":"b046cd77c79cc8e11522200053c9c9c9312de2b9196a0fcbf296104cfeeefcc5"} Oct 30 21:39:57.696089 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:57.696056 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bhdkg" event={"ID":"b8ed5eda-cbec-4adf-93b1-0ee0f1203f30","Type":"ContainerStarted","Data":"3c25529fba1c7992a7fe1e97c929f0efac84857e1db8a14cbbd06ff7a8e1760e"} Oct 30 21:39:57.696188 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:57.696111 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bhdkg" event={"ID":"b8ed5eda-cbec-4adf-93b1-0ee0f1203f30","Type":"ContainerStarted","Data":"eecd9db481580b49d59a74dcbc9a329ce9448bd64ee9ea4929683f64daf18753"} Oct 30 21:39:57.900571 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:57.900508 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:57.918012 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:57.917986 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6db5565879-ndj8n"] Oct 30 21:39:57.935344 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:57.935300 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6db5565879-ndj8n" Oct 30 21:39:57.935719 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:57.935693 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6db5565879-ndj8n"] Oct 30 21:39:57.941422 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:57.941371 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 30 21:39:57.943108 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:57.943056 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 30 21:39:57.943228 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:57.943171 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 30 21:39:57.943337 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:57.943322 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 21:39:57.943426 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:57.943370 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 21:39:57.943513 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:57.943499 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 30 21:39:57.943714 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:57.943673 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 30 21:39:57.943901 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:57.943885 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-f4ndj\"" Oct 30 21:39:58.024952 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:58.024917 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ec3fe241-c753-4845-8b7a-0469c73e4c3b-console-oauth-config\") pod \"console-6db5565879-ndj8n\" (UID: \"ec3fe241-c753-4845-8b7a-0469c73e4c3b\") " pod="openshift-console/console-6db5565879-ndj8n" Oct 30 21:39:58.025096 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:58.024959 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ec3fe241-c753-4845-8b7a-0469c73e4c3b-service-ca\") pod \"console-6db5565879-ndj8n\" (UID: \"ec3fe241-c753-4845-8b7a-0469c73e4c3b\") " pod="openshift-console/console-6db5565879-ndj8n" Oct 30 21:39:58.025096 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:58.024987 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ec3fe241-c753-4845-8b7a-0469c73e4c3b-oauth-serving-cert\") pod \"console-6db5565879-ndj8n\" (UID: \"ec3fe241-c753-4845-8b7a-0469c73e4c3b\") " pod="openshift-console/console-6db5565879-ndj8n" Oct 30 21:39:58.025096 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:58.025016 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rczmc\" (UniqueName: \"kubernetes.io/projected/ec3fe241-c753-4845-8b7a-0469c73e4c3b-kube-api-access-rczmc\") pod \"console-6db5565879-ndj8n\" (UID: \"ec3fe241-c753-4845-8b7a-0469c73e4c3b\") " pod="openshift-console/console-6db5565879-ndj8n" Oct 30 21:39:58.025096 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:58.025059 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ec3fe241-c753-4845-8b7a-0469c73e4c3b-console-serving-cert\") pod \"console-6db5565879-ndj8n\" (UID: \"ec3fe241-c753-4845-8b7a-0469c73e4c3b\") " pod="openshift-console/console-6db5565879-ndj8n" Oct 30 21:39:58.025096 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:58.025085 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ec3fe241-c753-4845-8b7a-0469c73e4c3b-console-config\") pod \"console-6db5565879-ndj8n\" (UID: \"ec3fe241-c753-4845-8b7a-0469c73e4c3b\") " pod="openshift-console/console-6db5565879-ndj8n" Oct 30 21:39:58.025300 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:58.025175 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ec3fe241-c753-4845-8b7a-0469c73e4c3b-tmp\") pod \"console-6db5565879-ndj8n\" (UID: \"ec3fe241-c753-4845-8b7a-0469c73e4c3b\") " pod="openshift-console/console-6db5565879-ndj8n" Oct 30 21:39:58.047114 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:58.047083 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:58.126196 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:58.126166 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ec3fe241-c753-4845-8b7a-0469c73e4c3b-tmp\") pod \"console-6db5565879-ndj8n\" (UID: \"ec3fe241-c753-4845-8b7a-0469c73e4c3b\") " pod="openshift-console/console-6db5565879-ndj8n" Oct 30 21:39:58.126356 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:58.126210 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ec3fe241-c753-4845-8b7a-0469c73e4c3b-console-oauth-config\") pod \"console-6db5565879-ndj8n\" (UID: \"ec3fe241-c753-4845-8b7a-0469c73e4c3b\") " pod="openshift-console/console-6db5565879-ndj8n" Oct 30 21:39:58.126356 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:58.126240 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ec3fe241-c753-4845-8b7a-0469c73e4c3b-service-ca\") pod \"console-6db5565879-ndj8n\" (UID: \"ec3fe241-c753-4845-8b7a-0469c73e4c3b\") " pod="openshift-console/console-6db5565879-ndj8n" Oct 30 21:39:58.126356 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:58.126281 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ec3fe241-c753-4845-8b7a-0469c73e4c3b-oauth-serving-cert\") pod \"console-6db5565879-ndj8n\" (UID: \"ec3fe241-c753-4845-8b7a-0469c73e4c3b\") " pod="openshift-console/console-6db5565879-ndj8n" Oct 30 21:39:58.126356 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:58.126303 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rczmc\" (UniqueName: \"kubernetes.io/projected/ec3fe241-c753-4845-8b7a-0469c73e4c3b-kube-api-access-rczmc\") pod \"console-6db5565879-ndj8n\" (UID: \"ec3fe241-c753-4845-8b7a-0469c73e4c3b\") " pod="openshift-console/console-6db5565879-ndj8n" Oct 30 21:39:58.126356 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:58.126340 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ec3fe241-c753-4845-8b7a-0469c73e4c3b-console-serving-cert\") pod \"console-6db5565879-ndj8n\" (UID: \"ec3fe241-c753-4845-8b7a-0469c73e4c3b\") " pod="openshift-console/console-6db5565879-ndj8n" Oct 30 21:39:58.126637 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:58.126366 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ec3fe241-c753-4845-8b7a-0469c73e4c3b-console-config\") pod \"console-6db5565879-ndj8n\" (UID: \"ec3fe241-c753-4845-8b7a-0469c73e4c3b\") " pod="openshift-console/console-6db5565879-ndj8n" Oct 30 21:39:58.126699 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:58.126636 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ec3fe241-c753-4845-8b7a-0469c73e4c3b-tmp\") pod \"console-6db5565879-ndj8n\" (UID: \"ec3fe241-c753-4845-8b7a-0469c73e4c3b\") " pod="openshift-console/console-6db5565879-ndj8n" Oct 30 21:39:58.127117 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:58.127095 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ec3fe241-c753-4845-8b7a-0469c73e4c3b-console-config\") pod \"console-6db5565879-ndj8n\" (UID: \"ec3fe241-c753-4845-8b7a-0469c73e4c3b\") " pod="openshift-console/console-6db5565879-ndj8n" Oct 30 21:39:58.127338 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:58.127299 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ec3fe241-c753-4845-8b7a-0469c73e4c3b-oauth-serving-cert\") pod \"console-6db5565879-ndj8n\" (UID: \"ec3fe241-c753-4845-8b7a-0469c73e4c3b\") " pod="openshift-console/console-6db5565879-ndj8n" Oct 30 21:39:58.127644 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:58.127609 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ec3fe241-c753-4845-8b7a-0469c73e4c3b-service-ca\") pod \"console-6db5565879-ndj8n\" (UID: \"ec3fe241-c753-4845-8b7a-0469c73e4c3b\") " pod="openshift-console/console-6db5565879-ndj8n" Oct 30 21:39:58.130838 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:58.130814 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ec3fe241-c753-4845-8b7a-0469c73e4c3b-console-serving-cert\") pod \"console-6db5565879-ndj8n\" (UID: \"ec3fe241-c753-4845-8b7a-0469c73e4c3b\") " pod="openshift-console/console-6db5565879-ndj8n" Oct 30 21:39:58.131108 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:58.131094 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ec3fe241-c753-4845-8b7a-0469c73e4c3b-console-oauth-config\") pod \"console-6db5565879-ndj8n\" (UID: \"ec3fe241-c753-4845-8b7a-0469c73e4c3b\") " pod="openshift-console/console-6db5565879-ndj8n" Oct 30 21:39:58.140402 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:58.140364 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rczmc\" (UniqueName: \"kubernetes.io/projected/ec3fe241-c753-4845-8b7a-0469c73e4c3b-kube-api-access-rczmc\") pod \"console-6db5565879-ndj8n\" (UID: \"ec3fe241-c753-4845-8b7a-0469c73e4c3b\") " pod="openshift-console/console-6db5565879-ndj8n" Oct 30 21:39:58.197576 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:58.197513 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:58.265841 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:58.265715 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6db5565879-ndj8n" Oct 30 21:39:58.339535 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:58.339495 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:58.476781 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:58.476712 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-m7qjt_8b0ede8c-1c7c-479c-8c04-29d849fdbf43/node-ca/0.log" Oct 30 21:39:58.601257 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:58.601224 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-hrkzk"] Oct 30 21:39:58.620800 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:58.619927 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6db5565879-ndj8n"] Oct 30 21:39:58.620800 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:58.619966 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-hrkzk"] Oct 30 21:39:58.620800 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:58.620061 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-hrkzk" Oct 30 21:39:58.620800 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:58.620246 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:58.625672 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:58.625641 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 21:39:58.625672 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:58.625669 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 21:39:58.625910 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:58.625892 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Oct 30 21:39:58.625997 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:58.625942 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-9shtx\"" Oct 30 21:39:58.626103 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:58.626085 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Oct 30 21:39:58.627022 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:58.627003 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 21:39:58.700690 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:58.700650 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6db5565879-ndj8n" event={"ID":"ec3fe241-c753-4845-8b7a-0469c73e4c3b","Type":"ContainerStarted","Data":"6c704cae0f9a5285742a852fc15009b450a18501da7c1d9946d7526ad8eae067"} Oct 30 21:39:58.703529 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:58.703501 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bhdkg" event={"ID":"b8ed5eda-cbec-4adf-93b1-0ee0f1203f30","Type":"ContainerStarted","Data":"e880d585ebe9cd4e256483695e163ee25574f2421853f20e30502dc1550151d8"} Oct 30 21:39:58.731302 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:58.730951 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/568ba29e-ccc3-4a4c-9a4f-192c0b931b0d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-hrkzk\" (UID: \"568ba29e-ccc3-4a4c-9a4f-192c0b931b0d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-hrkzk" Oct 30 21:39:58.731302 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:58.731001 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wv4r\" (UniqueName: \"kubernetes.io/projected/568ba29e-ccc3-4a4c-9a4f-192c0b931b0d-kube-api-access-5wv4r\") pod \"prometheus-operator-9cd89b74c-hrkzk\" (UID: \"568ba29e-ccc3-4a4c-9a4f-192c0b931b0d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-hrkzk" Oct 30 21:39:58.731302 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:58.731104 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/568ba29e-ccc3-4a4c-9a4f-192c0b931b0d-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-hrkzk\" (UID: \"568ba29e-ccc3-4a4c-9a4f-192c0b931b0d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-hrkzk" Oct 30 21:39:58.731302 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:58.731133 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/568ba29e-ccc3-4a4c-9a4f-192c0b931b0d-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-hrkzk\" (UID: \"568ba29e-ccc3-4a4c-9a4f-192c0b931b0d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-hrkzk" Oct 30 21:39:58.770417 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:58.770371 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:39:58.831799 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:58.831765 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/568ba29e-ccc3-4a4c-9a4f-192c0b931b0d-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-hrkzk\" (UID: \"568ba29e-ccc3-4a4c-9a4f-192c0b931b0d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-hrkzk" Oct 30 21:39:58.831964 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:58.831819 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/568ba29e-ccc3-4a4c-9a4f-192c0b931b0d-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-hrkzk\" (UID: \"568ba29e-ccc3-4a4c-9a4f-192c0b931b0d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-hrkzk" Oct 30 21:39:58.831964 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:58.831854 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/568ba29e-ccc3-4a4c-9a4f-192c0b931b0d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-hrkzk\" (UID: \"568ba29e-ccc3-4a4c-9a4f-192c0b931b0d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-hrkzk" Oct 30 21:39:58.831964 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:58.831887 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wv4r\" (UniqueName: \"kubernetes.io/projected/568ba29e-ccc3-4a4c-9a4f-192c0b931b0d-kube-api-access-5wv4r\") pod \"prometheus-operator-9cd89b74c-hrkzk\" (UID: \"568ba29e-ccc3-4a4c-9a4f-192c0b931b0d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-hrkzk" Oct 30 21:39:58.832578 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:58.832551 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/568ba29e-ccc3-4a4c-9a4f-192c0b931b0d-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-hrkzk\" (UID: \"568ba29e-ccc3-4a4c-9a4f-192c0b931b0d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-hrkzk" Oct 30 21:39:58.835013 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:58.834987 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/568ba29e-ccc3-4a4c-9a4f-192c0b931b0d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-hrkzk\" (UID: \"568ba29e-ccc3-4a4c-9a4f-192c0b931b0d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-hrkzk" Oct 30 21:39:58.835101 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:58.835015 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/568ba29e-ccc3-4a4c-9a4f-192c0b931b0d-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-hrkzk\" (UID: \"568ba29e-ccc3-4a4c-9a4f-192c0b931b0d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-hrkzk" Oct 30 21:39:58.844846 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:58.844820 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wv4r\" (UniqueName: \"kubernetes.io/projected/568ba29e-ccc3-4a4c-9a4f-192c0b931b0d-kube-api-access-5wv4r\") pod \"prometheus-operator-9cd89b74c-hrkzk\" (UID: \"568ba29e-ccc3-4a4c-9a4f-192c0b931b0d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-hrkzk" Oct 30 21:39:58.962916 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:58.962880 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-hrkzk" Oct 30 21:39:59.257203 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:59.257172 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-hrkzk"] Oct 30 21:39:59.261544 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:39:59.261515 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod568ba29e_ccc3_4a4c_9a4f_192c0b931b0d.slice/crio-628a1fbc58b9782013e5efba7cc6b3ac111a5174cfc6ea6086c187acca1344f4 WatchSource:0}: Error finding container 628a1fbc58b9782013e5efba7cc6b3ac111a5174cfc6ea6086c187acca1344f4: Status 404 returned error can't find the container with id 628a1fbc58b9782013e5efba7cc6b3ac111a5174cfc6ea6086c187acca1344f4 Oct 30 21:39:59.709012 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:59.708773 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-hrkzk" event={"ID":"568ba29e-ccc3-4a4c-9a4f-192c0b931b0d","Type":"ContainerStarted","Data":"628a1fbc58b9782013e5efba7cc6b3ac111a5174cfc6ea6086c187acca1344f4"} Oct 30 21:39:59.711631 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:59.711604 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-v9jlm" event={"ID":"e2c4958f-c7a5-4b6a-9b29-c7e2cd7d2d41","Type":"ContainerStarted","Data":"6759d169414e0c8bf22e3445f3b74664d3c7f38c8421e267fec4c801db4df435"} Oct 30 21:39:59.711756 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:59.711651 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-v9jlm" event={"ID":"e2c4958f-c7a5-4b6a-9b29-c7e2cd7d2d41","Type":"ContainerStarted","Data":"08a2659468f01248185d967363b36c3900dc1ef13c0200afa0db4a7ade523841"} Oct 30 21:39:59.711834 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:59.711771 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-v9jlm" Oct 30 21:39:59.742548 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:39:59.742497 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-v9jlm" podStartSLOduration=1.4309623519999999 podStartE2EDuration="3.742479769s" totalImagesPullingTime="2.311517417s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:59.740180626 +0000 UTC m=+37.947759640" watchObservedRunningTime="2026-10-30 21:39:59.742479769 +0000 UTC m=+37.950058927" Oct 30 21:40:00.604260 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:00.604168 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:00.715959 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:00.715920 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tj676" event={"ID":"3018f90d-07fd-4d10-80a3-1f14248e638c","Type":"ContainerStarted","Data":"d4612b4b53f12dd7a5f1580e3428b3928575e8bf856df9e2469c5c2eaa0f6bf1"} Oct 30 21:40:00.746305 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:00.746272 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:01.496796 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:01.496471 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:01.722036 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:01.721989 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-hrkzk" event={"ID":"568ba29e-ccc3-4a4c-9a4f-192c0b931b0d","Type":"ContainerStarted","Data":"84d3b1b62893df8cadc11d4ff48ac5eb3f838525f91fe75697fe90d2bdaf7806"} Oct 30 21:40:01.722734 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:01.722085 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-hrkzk" event={"ID":"568ba29e-ccc3-4a4c-9a4f-192c0b931b0d","Type":"ContainerStarted","Data":"da5809ded555a84627c38448fdf869cfe79c2cd067e32a511a844557223d0558"} Oct 30 21:40:01.724457 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:01.724420 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bhdkg" event={"ID":"b8ed5eda-cbec-4adf-93b1-0ee0f1203f30","Type":"ContainerStarted","Data":"09d4915102d36b127bbcdfd5a961b24fd2c53c8fecc5082a9fab95960abcf473"} Oct 30 21:40:01.743648 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:01.743587 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-tj676" podStartSLOduration=2.374977455 podStartE2EDuration="5.743575565s" totalImagesPullingTime="3.36859811s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:00.737631437 +0000 UTC m=+38.945210409" watchObservedRunningTime="2026-10-30 21:40:01.743575565 +0000 UTC m=+39.951154506" Oct 30 21:40:01.743836 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:01.743814 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-hrkzk" podStartSLOduration=1.910361186 podStartE2EDuration="3.743807457s" totalImagesPullingTime="1.833446271s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:01.743160357 +0000 UTC m=+39.950739316" watchObservedRunningTime="2026-10-30 21:40:01.743807457 +0000 UTC m=+39.951386413" Oct 30 21:40:01.767633 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:01.767556 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-bhdkg" podStartSLOduration=2.139392632 podStartE2EDuration="5.767543377s" totalImagesPullingTime="3.628150745s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:39:56 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:01.766283757 +0000 UTC m=+39.973862726" watchObservedRunningTime="2026-10-30 21:40:01.767543377 +0000 UTC m=+39.975122334" Oct 30 21:40:02.730116 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:02.730079 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6db5565879-ndj8n" event={"ID":"ec3fe241-c753-4845-8b7a-0469c73e4c3b","Type":"ContainerStarted","Data":"2a6e9dd3dd9adf00dada2b6d30b6309408993f8f70e9000cfb148ba4d85a30f5"} Oct 30 21:40:02.751672 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:02.751582 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6db5565879-ndj8n" podStartSLOduration=2.117343683 podStartE2EDuration="5.751565649s" totalImagesPullingTime="3.634221966s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:02.750980136 +0000 UTC m=+40.958559093" watchObservedRunningTime="2026-10-30 21:40:02.751565649 +0000 UTC m=+40.959144607" Oct 30 21:40:03.352575 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:03.352543 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:03.499154 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:03.499122 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:03.793992 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:03.793957 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:03.962400 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:03.962351 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:03.976539 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:03.976515 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-44hwx"] Oct 30 21:40:03.999566 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:03.999531 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-44hwx"] Oct 30 21:40:03.999686 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:03.999645 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-44hwx" Oct 30 21:40:04.008643 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:04.008615 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-4xbzf"] Oct 30 21:40:04.012331 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:04.012314 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 30 21:40:04.012331 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:04.012325 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-gd9hw\"" Oct 30 21:40:04.012331 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:04.012318 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 30 21:40:04.021883 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:04.021866 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4xbzf" Oct 30 21:40:04.022562 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:04.022541 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-hbg8k"] Oct 30 21:40:04.035847 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:04.035824 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-4xbzf"] Oct 30 21:40:04.035980 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:04.035957 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hbg8k" Oct 30 21:40:04.074017 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:04.073967 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/971a083e-69b8-46d1-b56a-9ccf0f4d557d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-44hwx\" (UID: \"971a083e-69b8-46d1-b56a-9ccf0f4d557d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-44hwx" Oct 30 21:40:04.074017 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:04.073999 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/971a083e-69b8-46d1-b56a-9ccf0f4d557d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-44hwx\" (UID: \"971a083e-69b8-46d1-b56a-9ccf0f4d557d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-44hwx" Oct 30 21:40:04.074168 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:04.074022 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfw8c\" (UniqueName: \"kubernetes.io/projected/971a083e-69b8-46d1-b56a-9ccf0f4d557d-kube-api-access-vfw8c\") pod \"openshift-state-metrics-7bd7489957-44hwx\" (UID: \"971a083e-69b8-46d1-b56a-9ccf0f4d557d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-44hwx" Oct 30 21:40:04.074168 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:04.074128 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/971a083e-69b8-46d1-b56a-9ccf0f4d557d-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-44hwx\" (UID: \"971a083e-69b8-46d1-b56a-9ccf0f4d557d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-44hwx" Oct 30 21:40:04.111630 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:04.111601 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 30 21:40:04.111760 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:04.111683 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 30 21:40:04.111760 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:04.111746 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-9f5cm\"" Oct 30 21:40:04.111867 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:04.111779 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 21:40:04.111909 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:04.111903 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 21:40:04.112124 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:04.112109 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 30 21:40:04.112169 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:04.112127 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 21:40:04.113111 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:04.113099 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-ls765\"" Oct 30 21:40:04.137220 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:04.137192 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:04.174918 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:04.174893 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/971a083e-69b8-46d1-b56a-9ccf0f4d557d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-44hwx\" (UID: \"971a083e-69b8-46d1-b56a-9ccf0f4d557d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-44hwx" Oct 30 21:40:04.175027 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:04.174922 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62-node-exporter-wtmp\") pod \"node-exporter-hbg8k\" (UID: \"c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62\") " pod="openshift-monitoring/node-exporter-hbg8k" Oct 30 21:40:04.175027 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:04.174943 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62-node-exporter-textfile\") pod \"node-exporter-hbg8k\" (UID: \"c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62\") " pod="openshift-monitoring/node-exporter-hbg8k" Oct 30 21:40:04.175027 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:04.174972 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfw8c\" (UniqueName: \"kubernetes.io/projected/971a083e-69b8-46d1-b56a-9ccf0f4d557d-kube-api-access-vfw8c\") pod \"openshift-state-metrics-7bd7489957-44hwx\" (UID: \"971a083e-69b8-46d1-b56a-9ccf0f4d557d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-44hwx" Oct 30 21:40:04.175152 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:04.175041 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62-node-exporter-accelerators-collector-config\") pod \"node-exporter-hbg8k\" (UID: \"c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62\") " pod="openshift-monitoring/node-exporter-hbg8k" Oct 30 21:40:04.175152 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:04.175092 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f4b104cb-8d0c-4519-86e0-07ea0e71a4a0-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-4xbzf\" (UID: \"f4b104cb-8d0c-4519-86e0-07ea0e71a4a0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4xbzf" Oct 30 21:40:04.175248 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:04.175230 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62-node-exporter-tls\") pod \"node-exporter-hbg8k\" (UID: \"c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62\") " pod="openshift-monitoring/node-exporter-hbg8k" Oct 30 21:40:04.175305 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:04.175264 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/971a083e-69b8-46d1-b56a-9ccf0f4d557d-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-44hwx\" (UID: \"971a083e-69b8-46d1-b56a-9ccf0f4d557d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-44hwx" Oct 30 21:40:04.175357 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:04.175316 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f4b104cb-8d0c-4519-86e0-07ea0e71a4a0-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-4xbzf\" (UID: \"f4b104cb-8d0c-4519-86e0-07ea0e71a4a0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4xbzf" Oct 30 21:40:04.175488 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:04.175473 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f4b104cb-8d0c-4519-86e0-07ea0e71a4a0-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-4xbzf\" (UID: \"f4b104cb-8d0c-4519-86e0-07ea0e71a4a0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4xbzf" Oct 30 21:40:04.175538 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:04.175506 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62-root\") pod \"node-exporter-hbg8k\" (UID: \"c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62\") " pod="openshift-monitoring/node-exporter-hbg8k" Oct 30 21:40:04.175583 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:04.175548 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/971a083e-69b8-46d1-b56a-9ccf0f4d557d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-44hwx\" (UID: \"971a083e-69b8-46d1-b56a-9ccf0f4d557d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-44hwx" Oct 30 21:40:04.175618 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:04.175581 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpmtf\" (UniqueName: \"kubernetes.io/projected/c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62-kube-api-access-qpmtf\") pod \"node-exporter-hbg8k\" (UID: \"c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62\") " pod="openshift-monitoring/node-exporter-hbg8k" Oct 30 21:40:04.175618 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:04.175599 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62-metrics-client-ca\") pod \"node-exporter-hbg8k\" (UID: \"c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62\") " pod="openshift-monitoring/node-exporter-hbg8k" Oct 30 21:40:04.175711 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:04.175621 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hbg8k\" (UID: \"c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62\") " pod="openshift-monitoring/node-exporter-hbg8k" Oct 30 21:40:04.175751 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:04.175695 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f4b104cb-8d0c-4519-86e0-07ea0e71a4a0-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-4xbzf\" (UID: \"f4b104cb-8d0c-4519-86e0-07ea0e71a4a0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4xbzf" Oct 30 21:40:04.175787 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:04.175751 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f4b104cb-8d0c-4519-86e0-07ea0e71a4a0-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-4xbzf\" (UID: \"f4b104cb-8d0c-4519-86e0-07ea0e71a4a0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4xbzf" Oct 30 21:40:04.175818 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:04.175798 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62-sys\") pod \"node-exporter-hbg8k\" (UID: \"c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62\") " pod="openshift-monitoring/node-exporter-hbg8k" Oct 30 21:40:04.175894 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:04.175878 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgq7t\" (UniqueName: \"kubernetes.io/projected/f4b104cb-8d0c-4519-86e0-07ea0e71a4a0-kube-api-access-wgq7t\") pod \"kube-state-metrics-5dc8d7d89b-4xbzf\" (UID: \"f4b104cb-8d0c-4519-86e0-07ea0e71a4a0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4xbzf" Oct 30 21:40:04.176084 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:04.176065 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/971a083e-69b8-46d1-b56a-9ccf0f4d557d-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-44hwx\" (UID: \"971a083e-69b8-46d1-b56a-9ccf0f4d557d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-44hwx" Oct 30 21:40:04.179048 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:04.179028 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/971a083e-69b8-46d1-b56a-9ccf0f4d557d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-44hwx\" (UID: \"971a083e-69b8-46d1-b56a-9ccf0f4d557d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-44hwx" Oct 30 21:40:04.179152 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:04.179054 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/971a083e-69b8-46d1-b56a-9ccf0f4d557d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-44hwx\" (UID: \"971a083e-69b8-46d1-b56a-9ccf0f4d557d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-44hwx" Oct 30 21:40:04.188227 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:04.188202 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfw8c\" (UniqueName: \"kubernetes.io/projected/971a083e-69b8-46d1-b56a-9ccf0f4d557d-kube-api-access-vfw8c\") pod \"openshift-state-metrics-7bd7489957-44hwx\" (UID: \"971a083e-69b8-46d1-b56a-9ccf0f4d557d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-44hwx" Oct 30 21:40:04.276312 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:04.276285 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgq7t\" (UniqueName: \"kubernetes.io/projected/f4b104cb-8d0c-4519-86e0-07ea0e71a4a0-kube-api-access-wgq7t\") pod \"kube-state-metrics-5dc8d7d89b-4xbzf\" (UID: \"f4b104cb-8d0c-4519-86e0-07ea0e71a4a0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4xbzf" Oct 30 21:40:04.276440 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:04.276315 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62-node-exporter-wtmp\") pod \"node-exporter-hbg8k\" (UID: \"c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62\") " pod="openshift-monitoring/node-exporter-hbg8k" Oct 30 21:40:04.276440 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:04.276335 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62-node-exporter-textfile\") pod \"node-exporter-hbg8k\" (UID: \"c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62\") " pod="openshift-monitoring/node-exporter-hbg8k" Oct 30 21:40:04.276440 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:04.276356 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62-node-exporter-accelerators-collector-config\") pod \"node-exporter-hbg8k\" (UID: \"c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62\") " pod="openshift-monitoring/node-exporter-hbg8k" Oct 30 21:40:04.276622 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:04.276432 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f4b104cb-8d0c-4519-86e0-07ea0e71a4a0-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-4xbzf\" (UID: \"f4b104cb-8d0c-4519-86e0-07ea0e71a4a0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4xbzf" Oct 30 21:40:04.276622 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:04.276488 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62-node-exporter-wtmp\") pod \"node-exporter-hbg8k\" (UID: \"c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62\") " pod="openshift-monitoring/node-exporter-hbg8k" Oct 30 21:40:04.276622 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:04.276515 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62-node-exporter-tls\") pod \"node-exporter-hbg8k\" (UID: \"c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62\") " pod="openshift-monitoring/node-exporter-hbg8k" Oct 30 21:40:04.276622 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:04.276548 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f4b104cb-8d0c-4519-86e0-07ea0e71a4a0-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-4xbzf\" (UID: \"f4b104cb-8d0c-4519-86e0-07ea0e71a4a0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4xbzf" Oct 30 21:40:04.276622 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:04.276591 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f4b104cb-8d0c-4519-86e0-07ea0e71a4a0-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-4xbzf\" (UID: \"f4b104cb-8d0c-4519-86e0-07ea0e71a4a0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4xbzf" Oct 30 21:40:04.276857 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:04.276663 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62-root\") pod \"node-exporter-hbg8k\" (UID: \"c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62\") " pod="openshift-monitoring/node-exporter-hbg8k" Oct 30 21:40:04.276857 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:04.276775 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62-node-exporter-textfile\") pod \"node-exporter-hbg8k\" (UID: \"c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62\") " pod="openshift-monitoring/node-exporter-hbg8k" Oct 30 21:40:04.276857 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:04.276847 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpmtf\" (UniqueName: \"kubernetes.io/projected/c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62-kube-api-access-qpmtf\") pod \"node-exporter-hbg8k\" (UID: \"c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62\") " pod="openshift-monitoring/node-exporter-hbg8k" Oct 30 21:40:04.276994 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:04.276881 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62-metrics-client-ca\") pod \"node-exporter-hbg8k\" (UID: \"c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62\") " pod="openshift-monitoring/node-exporter-hbg8k" Oct 30 21:40:04.276994 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:04.276912 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hbg8k\" (UID: \"c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62\") " pod="openshift-monitoring/node-exporter-hbg8k" Oct 30 21:40:04.276994 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:04.276950 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f4b104cb-8d0c-4519-86e0-07ea0e71a4a0-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-4xbzf\" (UID: \"f4b104cb-8d0c-4519-86e0-07ea0e71a4a0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4xbzf" Oct 30 21:40:04.276994 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:04.276978 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f4b104cb-8d0c-4519-86e0-07ea0e71a4a0-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-4xbzf\" (UID: \"f4b104cb-8d0c-4519-86e0-07ea0e71a4a0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4xbzf" Oct 30 21:40:04.277179 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:04.277008 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62-sys\") pod \"node-exporter-hbg8k\" (UID: \"c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62\") " pod="openshift-monitoring/node-exporter-hbg8k" Oct 30 21:40:04.277179 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:04.277006 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62-root\") pod \"node-exporter-hbg8k\" (UID: \"c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62\") " pod="openshift-monitoring/node-exporter-hbg8k" Oct 30 21:40:04.277179 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:04.277052 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62-node-exporter-accelerators-collector-config\") pod \"node-exporter-hbg8k\" (UID: \"c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62\") " pod="openshift-monitoring/node-exporter-hbg8k" Oct 30 21:40:04.277179 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:04.277079 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62-sys\") pod \"node-exporter-hbg8k\" (UID: \"c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62\") " pod="openshift-monitoring/node-exporter-hbg8k" Oct 30 21:40:04.277677 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:04.277374 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f4b104cb-8d0c-4519-86e0-07ea0e71a4a0-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-4xbzf\" (UID: \"f4b104cb-8d0c-4519-86e0-07ea0e71a4a0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4xbzf" Oct 30 21:40:04.277677 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:04.277620 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f4b104cb-8d0c-4519-86e0-07ea0e71a4a0-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-4xbzf\" (UID: \"f4b104cb-8d0c-4519-86e0-07ea0e71a4a0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4xbzf" Oct 30 21:40:04.277890 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:04.277870 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62-metrics-client-ca\") pod \"node-exporter-hbg8k\" (UID: \"c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62\") " pod="openshift-monitoring/node-exporter-hbg8k" Oct 30 21:40:04.277945 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:04.277897 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f4b104cb-8d0c-4519-86e0-07ea0e71a4a0-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-4xbzf\" (UID: \"f4b104cb-8d0c-4519-86e0-07ea0e71a4a0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4xbzf" Oct 30 21:40:04.279145 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:04.279118 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f4b104cb-8d0c-4519-86e0-07ea0e71a4a0-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-4xbzf\" (UID: \"f4b104cb-8d0c-4519-86e0-07ea0e71a4a0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4xbzf" Oct 30 21:40:04.279295 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:04.279266 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62-node-exporter-tls\") pod \"node-exporter-hbg8k\" (UID: \"c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62\") " pod="openshift-monitoring/node-exporter-hbg8k" Oct 30 21:40:04.279554 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:04.279540 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f4b104cb-8d0c-4519-86e0-07ea0e71a4a0-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-4xbzf\" (UID: \"f4b104cb-8d0c-4519-86e0-07ea0e71a4a0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4xbzf" Oct 30 21:40:04.280013 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:04.279999 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hbg8k\" (UID: \"c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62\") " pod="openshift-monitoring/node-exporter-hbg8k" Oct 30 21:40:04.289704 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:04.289681 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgq7t\" (UniqueName: \"kubernetes.io/projected/f4b104cb-8d0c-4519-86e0-07ea0e71a4a0-kube-api-access-wgq7t\") pod \"kube-state-metrics-5dc8d7d89b-4xbzf\" (UID: \"f4b104cb-8d0c-4519-86e0-07ea0e71a4a0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4xbzf" Oct 30 21:40:04.289781 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:04.289762 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpmtf\" (UniqueName: \"kubernetes.io/projected/c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62-kube-api-access-qpmtf\") pod \"node-exporter-hbg8k\" (UID: \"c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62\") " pod="openshift-monitoring/node-exporter-hbg8k" Oct 30 21:40:04.338855 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:04.338802 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:04.350743 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:04.350723 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-44hwx" Oct 30 21:40:04.421284 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:04.421252 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hbg8k" Oct 30 21:40:04.443731 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:04.443567 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4xbzf" Oct 30 21:40:04.448467 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:40:04.448433 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7cdfe5f_5f27_41a4_87de_7f5f6d2c1d62.slice/crio-838254bb88a8cb6c96ecff274a46f301292ac57adff9a00097693925841d5b2d WatchSource:0}: Error finding container 838254bb88a8cb6c96ecff274a46f301292ac57adff9a00097693925841d5b2d: Status 404 returned error can't find the container with id 838254bb88a8cb6c96ecff274a46f301292ac57adff9a00097693925841d5b2d Oct 30 21:40:04.564922 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:04.564890 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:04.664110 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:04.663960 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-44hwx"] Oct 30 21:40:04.667009 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:40:04.666976 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod971a083e_69b8_46d1_b56a_9ccf0f4d557d.slice/crio-070283f63c96b42654c4445147ef147ff242174d1a6f64f36fc541812377392e WatchSource:0}: Error finding container 070283f63c96b42654c4445147ef147ff242174d1a6f64f36fc541812377392e: Status 404 returned error can't find the container with id 070283f63c96b42654c4445147ef147ff242174d1a6f64f36fc541812377392e Oct 30 21:40:04.736931 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:04.736903 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-44hwx" event={"ID":"971a083e-69b8-46d1-b56a-9ccf0f4d557d","Type":"ContainerStarted","Data":"070283f63c96b42654c4445147ef147ff242174d1a6f64f36fc541812377392e"} Oct 30 21:40:04.737976 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:04.737957 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hbg8k" event={"ID":"c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62","Type":"ContainerStarted","Data":"838254bb88a8cb6c96ecff274a46f301292ac57adff9a00097693925841d5b2d"} Oct 30 21:40:04.746723 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:04.746703 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-4xbzf"] Oct 30 21:40:04.749472 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:40:04.749448 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4b104cb_8d0c_4519_86e0_07ea0e71a4a0.slice/crio-042516063e94ac1953c1bab8314d5b5ef0ee01e12f473016848591f1489a8a69 WatchSource:0}: Error finding container 042516063e94ac1953c1bab8314d5b5ef0ee01e12f473016848591f1489a8a69: Status 404 returned error can't find the container with id 042516063e94ac1953c1bab8314d5b5ef0ee01e12f473016848591f1489a8a69 Oct 30 21:40:04.756901 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:04.756872 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:04.902673 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:04.902642 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:05.088128 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:05.088055 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:05.313112 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:05.313074 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:05.744351 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:05.744323 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-44hwx" event={"ID":"971a083e-69b8-46d1-b56a-9ccf0f4d557d","Type":"ContainerStarted","Data":"b9685c99384e2372977e15a879388452592f0a358e16490e7288889b633d43b0"} Oct 30 21:40:05.744537 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:05.744404 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-44hwx" event={"ID":"971a083e-69b8-46d1-b56a-9ccf0f4d557d","Type":"ContainerStarted","Data":"d5c6a1f727f65ba0b7abd08030bcd1df7ecb48b03534341a5de2096532b5bf95"} Oct 30 21:40:05.745714 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:05.745687 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4xbzf" event={"ID":"f4b104cb-8d0c-4519-86e0-07ea0e71a4a0","Type":"ContainerStarted","Data":"042516063e94ac1953c1bab8314d5b5ef0ee01e12f473016848591f1489a8a69"} Oct 30 21:40:05.748268 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:05.748245 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hbg8k" event={"ID":"c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62","Type":"ContainerStarted","Data":"3c28e35963f034e787dd1415b9ea422cbe50c5cf9343a50b978b6f619bc1ae98"} Oct 30 21:40:06.080254 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:06.080231 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6fd8fdc4c5-xlslr"] Oct 30 21:40:06.082719 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:06.082696 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6fd8fdc4c5-xlslr" Oct 30 21:40:06.091552 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:06.091529 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 30 21:40:06.091552 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:06.091546 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 30 21:40:06.093075 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:06.093057 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-3d1aq256tb889\"" Oct 30 21:40:06.093158 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:06.093086 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-g4b65\"" Oct 30 21:40:06.093158 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:06.093087 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 30 21:40:06.093158 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:06.093098 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 30 21:40:06.093158 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:06.093117 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 30 21:40:06.101674 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:06.101590 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6fd8fdc4c5-xlslr"] Oct 30 21:40:06.193098 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:06.191829 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/90136025-abf7-4dc6-ac4b-c3050fc417f6-tmp\") pod \"thanos-querier-6fd8fdc4c5-xlslr\" (UID: \"90136025-abf7-4dc6-ac4b-c3050fc417f6\") " pod="openshift-monitoring/thanos-querier-6fd8fdc4c5-xlslr" Oct 30 21:40:06.193098 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:06.191860 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/90136025-abf7-4dc6-ac4b-c3050fc417f6-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6fd8fdc4c5-xlslr\" (UID: \"90136025-abf7-4dc6-ac4b-c3050fc417f6\") " pod="openshift-monitoring/thanos-querier-6fd8fdc4c5-xlslr" Oct 30 21:40:06.193098 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:06.191885 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/90136025-abf7-4dc6-ac4b-c3050fc417f6-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6fd8fdc4c5-xlslr\" (UID: \"90136025-abf7-4dc6-ac4b-c3050fc417f6\") " pod="openshift-monitoring/thanos-querier-6fd8fdc4c5-xlslr" Oct 30 21:40:06.193098 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:06.191901 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/90136025-abf7-4dc6-ac4b-c3050fc417f6-secret-thanos-querier-tls\") pod \"thanos-querier-6fd8fdc4c5-xlslr\" (UID: \"90136025-abf7-4dc6-ac4b-c3050fc417f6\") " pod="openshift-monitoring/thanos-querier-6fd8fdc4c5-xlslr" Oct 30 21:40:06.193098 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:06.191918 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/90136025-abf7-4dc6-ac4b-c3050fc417f6-secret-grpc-tls\") pod \"thanos-querier-6fd8fdc4c5-xlslr\" (UID: \"90136025-abf7-4dc6-ac4b-c3050fc417f6\") " pod="openshift-monitoring/thanos-querier-6fd8fdc4c5-xlslr" Oct 30 21:40:06.193098 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:06.191942 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/90136025-abf7-4dc6-ac4b-c3050fc417f6-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6fd8fdc4c5-xlslr\" (UID: \"90136025-abf7-4dc6-ac4b-c3050fc417f6\") " pod="openshift-monitoring/thanos-querier-6fd8fdc4c5-xlslr" Oct 30 21:40:06.193098 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:06.191973 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/90136025-abf7-4dc6-ac4b-c3050fc417f6-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6fd8fdc4c5-xlslr\" (UID: \"90136025-abf7-4dc6-ac4b-c3050fc417f6\") " pod="openshift-monitoring/thanos-querier-6fd8fdc4c5-xlslr" Oct 30 21:40:06.193098 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:06.191991 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/90136025-abf7-4dc6-ac4b-c3050fc417f6-metrics-client-ca\") pod \"thanos-querier-6fd8fdc4c5-xlslr\" (UID: \"90136025-abf7-4dc6-ac4b-c3050fc417f6\") " pod="openshift-monitoring/thanos-querier-6fd8fdc4c5-xlslr" Oct 30 21:40:06.193098 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:06.192004 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qchq\" (UniqueName: \"kubernetes.io/projected/90136025-abf7-4dc6-ac4b-c3050fc417f6-kube-api-access-6qchq\") pod \"thanos-querier-6fd8fdc4c5-xlslr\" (UID: \"90136025-abf7-4dc6-ac4b-c3050fc417f6\") " pod="openshift-monitoring/thanos-querier-6fd8fdc4c5-xlslr" Oct 30 21:40:06.292722 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:06.292687 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/90136025-abf7-4dc6-ac4b-c3050fc417f6-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6fd8fdc4c5-xlslr\" (UID: \"90136025-abf7-4dc6-ac4b-c3050fc417f6\") " pod="openshift-monitoring/thanos-querier-6fd8fdc4c5-xlslr" Oct 30 21:40:06.292722 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:06.292723 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/90136025-abf7-4dc6-ac4b-c3050fc417f6-metrics-client-ca\") pod \"thanos-querier-6fd8fdc4c5-xlslr\" (UID: \"90136025-abf7-4dc6-ac4b-c3050fc417f6\") " pod="openshift-monitoring/thanos-querier-6fd8fdc4c5-xlslr" Oct 30 21:40:06.292924 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:06.292756 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qchq\" (UniqueName: \"kubernetes.io/projected/90136025-abf7-4dc6-ac4b-c3050fc417f6-kube-api-access-6qchq\") pod \"thanos-querier-6fd8fdc4c5-xlslr\" (UID: \"90136025-abf7-4dc6-ac4b-c3050fc417f6\") " pod="openshift-monitoring/thanos-querier-6fd8fdc4c5-xlslr" Oct 30 21:40:06.292924 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:06.292817 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/90136025-abf7-4dc6-ac4b-c3050fc417f6-tmp\") pod \"thanos-querier-6fd8fdc4c5-xlslr\" (UID: \"90136025-abf7-4dc6-ac4b-c3050fc417f6\") " pod="openshift-monitoring/thanos-querier-6fd8fdc4c5-xlslr" Oct 30 21:40:06.292924 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:06.292843 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/90136025-abf7-4dc6-ac4b-c3050fc417f6-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6fd8fdc4c5-xlslr\" (UID: \"90136025-abf7-4dc6-ac4b-c3050fc417f6\") " pod="openshift-monitoring/thanos-querier-6fd8fdc4c5-xlslr" Oct 30 21:40:06.293021 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:06.292932 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/90136025-abf7-4dc6-ac4b-c3050fc417f6-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6fd8fdc4c5-xlslr\" (UID: \"90136025-abf7-4dc6-ac4b-c3050fc417f6\") " pod="openshift-monitoring/thanos-querier-6fd8fdc4c5-xlslr" Oct 30 21:40:06.293021 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:06.292958 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/90136025-abf7-4dc6-ac4b-c3050fc417f6-secret-thanos-querier-tls\") pod \"thanos-querier-6fd8fdc4c5-xlslr\" (UID: \"90136025-abf7-4dc6-ac4b-c3050fc417f6\") " pod="openshift-monitoring/thanos-querier-6fd8fdc4c5-xlslr" Oct 30 21:40:06.293304 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:06.293270 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/90136025-abf7-4dc6-ac4b-c3050fc417f6-tmp\") pod \"thanos-querier-6fd8fdc4c5-xlslr\" (UID: \"90136025-abf7-4dc6-ac4b-c3050fc417f6\") " pod="openshift-monitoring/thanos-querier-6fd8fdc4c5-xlslr" Oct 30 21:40:06.293402 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:06.293336 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/90136025-abf7-4dc6-ac4b-c3050fc417f6-secret-grpc-tls\") pod \"thanos-querier-6fd8fdc4c5-xlslr\" (UID: \"90136025-abf7-4dc6-ac4b-c3050fc417f6\") " pod="openshift-monitoring/thanos-querier-6fd8fdc4c5-xlslr" Oct 30 21:40:06.293457 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:06.293414 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/90136025-abf7-4dc6-ac4b-c3050fc417f6-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6fd8fdc4c5-xlslr\" (UID: \"90136025-abf7-4dc6-ac4b-c3050fc417f6\") " pod="openshift-monitoring/thanos-querier-6fd8fdc4c5-xlslr" Oct 30 21:40:06.293584 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:06.293511 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/90136025-abf7-4dc6-ac4b-c3050fc417f6-metrics-client-ca\") pod \"thanos-querier-6fd8fdc4c5-xlslr\" (UID: \"90136025-abf7-4dc6-ac4b-c3050fc417f6\") " pod="openshift-monitoring/thanos-querier-6fd8fdc4c5-xlslr" Oct 30 21:40:06.296621 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:06.296597 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/90136025-abf7-4dc6-ac4b-c3050fc417f6-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6fd8fdc4c5-xlslr\" (UID: \"90136025-abf7-4dc6-ac4b-c3050fc417f6\") " pod="openshift-monitoring/thanos-querier-6fd8fdc4c5-xlslr" Oct 30 21:40:06.296760 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:06.296601 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/90136025-abf7-4dc6-ac4b-c3050fc417f6-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6fd8fdc4c5-xlslr\" (UID: \"90136025-abf7-4dc6-ac4b-c3050fc417f6\") " pod="openshift-monitoring/thanos-querier-6fd8fdc4c5-xlslr" Oct 30 21:40:06.296925 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:06.296896 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/90136025-abf7-4dc6-ac4b-c3050fc417f6-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6fd8fdc4c5-xlslr\" (UID: \"90136025-abf7-4dc6-ac4b-c3050fc417f6\") " pod="openshift-monitoring/thanos-querier-6fd8fdc4c5-xlslr" Oct 30 21:40:06.297088 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:06.296926 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/90136025-abf7-4dc6-ac4b-c3050fc417f6-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6fd8fdc4c5-xlslr\" (UID: \"90136025-abf7-4dc6-ac4b-c3050fc417f6\") " pod="openshift-monitoring/thanos-querier-6fd8fdc4c5-xlslr" Oct 30 21:40:06.297259 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:06.297166 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/90136025-abf7-4dc6-ac4b-c3050fc417f6-secret-grpc-tls\") pod \"thanos-querier-6fd8fdc4c5-xlslr\" (UID: \"90136025-abf7-4dc6-ac4b-c3050fc417f6\") " pod="openshift-monitoring/thanos-querier-6fd8fdc4c5-xlslr" Oct 30 21:40:06.297546 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:06.297525 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/90136025-abf7-4dc6-ac4b-c3050fc417f6-secret-thanos-querier-tls\") pod \"thanos-querier-6fd8fdc4c5-xlslr\" (UID: \"90136025-abf7-4dc6-ac4b-c3050fc417f6\") " pod="openshift-monitoring/thanos-querier-6fd8fdc4c5-xlslr" Oct 30 21:40:06.306189 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:06.306140 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qchq\" (UniqueName: \"kubernetes.io/projected/90136025-abf7-4dc6-ac4b-c3050fc417f6-kube-api-access-6qchq\") pod \"thanos-querier-6fd8fdc4c5-xlslr\" (UID: \"90136025-abf7-4dc6-ac4b-c3050fc417f6\") " pod="openshift-monitoring/thanos-querier-6fd8fdc4c5-xlslr" Oct 30 21:40:06.413244 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:06.413210 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6fd8fdc4c5-xlslr" Oct 30 21:40:06.731711 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:06.731576 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6fd8fdc4c5-xlslr"] Oct 30 21:40:06.733716 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:40:06.733695 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90136025_abf7_4dc6_ac4b_c3050fc417f6.slice/crio-787d69f12c43edd684ab02c16bca74f4532a0e8ad037dc6a52df731daa0d8246 WatchSource:0}: Error finding container 787d69f12c43edd684ab02c16bca74f4532a0e8ad037dc6a52df731daa0d8246: Status 404 returned error can't find the container with id 787d69f12c43edd684ab02c16bca74f4532a0e8ad037dc6a52df731daa0d8246 Oct 30 21:40:06.752696 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:06.752670 2574 generic.go:356] "Generic (PLEG): container finished" podID="c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62" containerID="3c28e35963f034e787dd1415b9ea422cbe50c5cf9343a50b978b6f619bc1ae98" exitCode=0 Oct 30 21:40:06.752823 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:06.752729 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hbg8k" event={"ID":"c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62","Type":"ContainerDied","Data":"3c28e35963f034e787dd1415b9ea422cbe50c5cf9343a50b978b6f619bc1ae98"} Oct 30 21:40:06.754087 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:06.754060 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6fd8fdc4c5-xlslr" event={"ID":"90136025-abf7-4dc6-ac4b-c3050fc417f6","Type":"ContainerStarted","Data":"787d69f12c43edd684ab02c16bca74f4532a0e8ad037dc6a52df731daa0d8246"} Oct 30 21:40:06.756272 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:06.756240 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-44hwx" event={"ID":"971a083e-69b8-46d1-b56a-9ccf0f4d557d","Type":"ContainerStarted","Data":"426fa9bcc788e727ae9c47d40670ef29122844e97846e34805de210798367613"} Oct 30 21:40:06.758324 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:06.758298 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4xbzf" event={"ID":"f4b104cb-8d0c-4519-86e0-07ea0e71a4a0","Type":"ContainerStarted","Data":"8a3018de5daa7890c43d97278483898bb953b55870bece5d5f6e7b72ceb9b366"} Oct 30 21:40:06.758441 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:06.758354 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4xbzf" event={"ID":"f4b104cb-8d0c-4519-86e0-07ea0e71a4a0","Type":"ContainerStarted","Data":"02c1690d0a8bb2420b1b6d314feed58ed0b3e254a46639ef4636c577c9e3c837"} Oct 30 21:40:06.758441 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:06.758398 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4xbzf" event={"ID":"f4b104cb-8d0c-4519-86e0-07ea0e71a4a0","Type":"ContainerStarted","Data":"7989e5e9337a1268d5cd91646a746d5a3f1d7977e7e2bb92ceccd95689430555"} Oct 30 21:40:06.815530 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:06.815477 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4xbzf" podStartSLOduration=2.5165998160000003 podStartE2EDuration="3.815459581s" totalImagesPullingTime="1.298859765s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:06.815085683 +0000 UTC m=+45.022664642" watchObservedRunningTime="2026-10-30 21:40:06.815459581 +0000 UTC m=+45.023038539" Oct 30 21:40:07.206580 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:07.206552 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:07.409717 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:07.409687 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:07.765021 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:07.764987 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hbg8k" event={"ID":"c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62","Type":"ContainerStarted","Data":"1dbf7b0f35aa3c019427b47b52a99ad48c26ab465a07baaf04715a82dbd16145"} Oct 30 21:40:07.765196 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:07.765046 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hbg8k" event={"ID":"c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62","Type":"ContainerStarted","Data":"6b2b9edcbb839da4805758bab92bd1a4a5c2e23ab611c3e3740e98b584592e6e"} Oct 30 21:40:07.796055 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:07.796012 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-44hwx" podStartSLOduration=3.564225337 podStartE2EDuration="4.795998821s" totalImagesPullingTime="1.231773484s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:06.869376112 +0000 UTC m=+45.076955069" watchObservedRunningTime="2026-10-30 21:40:07.795998821 +0000 UTC m=+46.003577779" Oct 30 21:40:08.267084 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:08.267047 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6db5565879-ndj8n" Oct 30 21:40:08.267528 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:08.267144 2574 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6db5565879-ndj8n" Oct 30 21:40:08.273154 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:08.273087 2574 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6db5565879-ndj8n" Oct 30 21:40:08.310527 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:08.309506 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-hbg8k" podStartSLOduration=4.069266098 podStartE2EDuration="5.309490593s" totalImagesPullingTime="1.151762881s" totalInitContainerRuntime="88.461614ms" isStatefulPod=true podCreationTimestamp="2026-10-30 21:40:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:07.796751715 +0000 UTC m=+46.004330672" watchObservedRunningTime="2026-10-30 21:40:08.309490593 +0000 UTC m=+46.517069551" Oct 30 21:40:08.515755 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:08.515719 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:08.579518 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:08.579494 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-c74454f78-xph4m"] Oct 30 21:40:08.581933 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:08.581918 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-c74454f78-xph4m" Oct 30 21:40:08.597963 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:08.597943 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 30 21:40:08.598052 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:08.597995 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 21:40:08.604034 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:08.604010 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-c74454f78-xph4m"] Oct 30 21:40:08.626171 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:08.626150 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-fhou7149cukh2\"" Oct 30 21:40:08.627258 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:08.627240 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-9cpg6\"" Oct 30 21:40:08.627324 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:08.627272 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 30 21:40:08.627324 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:08.627302 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 30 21:40:08.716119 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:08.716092 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d-client-ca-bundle\") pod \"metrics-server-c74454f78-xph4m\" (UID: \"df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d\") " pod="openshift-monitoring/metrics-server-c74454f78-xph4m" Oct 30 21:40:08.716202 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:08.716148 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d-metrics-server-audit-profiles\") pod \"metrics-server-c74454f78-xph4m\" (UID: \"df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d\") " pod="openshift-monitoring/metrics-server-c74454f78-xph4m" Oct 30 21:40:08.716252 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:08.716215 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-c74454f78-xph4m\" (UID: \"df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d\") " pod="openshift-monitoring/metrics-server-c74454f78-xph4m" Oct 30 21:40:08.716283 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:08.716255 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d-audit-log\") pod \"metrics-server-c74454f78-xph4m\" (UID: \"df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d\") " pod="openshift-monitoring/metrics-server-c74454f78-xph4m" Oct 30 21:40:08.716283 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:08.716272 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvrrj\" (UniqueName: \"kubernetes.io/projected/df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d-kube-api-access-bvrrj\") pod \"metrics-server-c74454f78-xph4m\" (UID: \"df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d\") " pod="openshift-monitoring/metrics-server-c74454f78-xph4m" Oct 30 21:40:08.716452 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:08.716434 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d-secret-metrics-server-client-certs\") pod \"metrics-server-c74454f78-xph4m\" (UID: \"df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d\") " pod="openshift-monitoring/metrics-server-c74454f78-xph4m" Oct 30 21:40:08.716522 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:08.716481 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d-secret-metrics-server-tls\") pod \"metrics-server-c74454f78-xph4m\" (UID: \"df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d\") " pod="openshift-monitoring/metrics-server-c74454f78-xph4m" Oct 30 21:40:08.770997 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:08.770965 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6fd8fdc4c5-xlslr" event={"ID":"90136025-abf7-4dc6-ac4b-c3050fc417f6","Type":"ContainerStarted","Data":"36c83c05bb4f064605457f2adab521428236da7cbd360048d75e96a69f28146e"} Oct 30 21:40:08.771160 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:08.771042 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6fd8fdc4c5-xlslr" event={"ID":"90136025-abf7-4dc6-ac4b-c3050fc417f6","Type":"ContainerStarted","Data":"45b18642546ce49fbc8bfd015b03741f7e23dfa0d8575e78a2fbf4dbeacc06e6"} Oct 30 21:40:08.771160 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:08.771083 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6fd8fdc4c5-xlslr" event={"ID":"90136025-abf7-4dc6-ac4b-c3050fc417f6","Type":"ContainerStarted","Data":"595028c3491a15dad32c6d3f31f1fc33bcddd3c6c6fcdfce9d3bbe7c0414a6ca"} Oct 30 21:40:08.775025 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:08.775003 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6db5565879-ndj8n" Oct 30 21:40:08.817057 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:08.816986 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d-metrics-server-audit-profiles\") pod \"metrics-server-c74454f78-xph4m\" (UID: \"df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d\") " pod="openshift-monitoring/metrics-server-c74454f78-xph4m" Oct 30 21:40:08.817057 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:08.817032 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-c74454f78-xph4m\" (UID: \"df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d\") " pod="openshift-monitoring/metrics-server-c74454f78-xph4m" Oct 30 21:40:08.817277 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:08.817079 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d-audit-log\") pod \"metrics-server-c74454f78-xph4m\" (UID: \"df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d\") " pod="openshift-monitoring/metrics-server-c74454f78-xph4m" Oct 30 21:40:08.817277 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:08.817112 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvrrj\" (UniqueName: \"kubernetes.io/projected/df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d-kube-api-access-bvrrj\") pod \"metrics-server-c74454f78-xph4m\" (UID: \"df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d\") " pod="openshift-monitoring/metrics-server-c74454f78-xph4m" Oct 30 21:40:08.817277 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:08.817203 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d-secret-metrics-server-client-certs\") pod \"metrics-server-c74454f78-xph4m\" (UID: \"df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d\") " pod="openshift-monitoring/metrics-server-c74454f78-xph4m" Oct 30 21:40:08.817277 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:08.817239 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d-secret-metrics-server-tls\") pod \"metrics-server-c74454f78-xph4m\" (UID: \"df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d\") " pod="openshift-monitoring/metrics-server-c74454f78-xph4m" Oct 30 21:40:08.817277 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:08.817276 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d-client-ca-bundle\") pod \"metrics-server-c74454f78-xph4m\" (UID: \"df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d\") " pod="openshift-monitoring/metrics-server-c74454f78-xph4m" Oct 30 21:40:08.817974 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:08.817940 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-c74454f78-xph4m\" (UID: \"df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d\") " pod="openshift-monitoring/metrics-server-c74454f78-xph4m" Oct 30 21:40:08.818068 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:08.818043 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d-metrics-server-audit-profiles\") pod \"metrics-server-c74454f78-xph4m\" (UID: \"df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d\") " pod="openshift-monitoring/metrics-server-c74454f78-xph4m" Oct 30 21:40:08.818132 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:08.818114 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d-audit-log\") pod \"metrics-server-c74454f78-xph4m\" (UID: \"df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d\") " pod="openshift-monitoring/metrics-server-c74454f78-xph4m" Oct 30 21:40:08.821434 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:08.821407 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d-secret-metrics-server-tls\") pod \"metrics-server-c74454f78-xph4m\" (UID: \"df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d\") " pod="openshift-monitoring/metrics-server-c74454f78-xph4m" Oct 30 21:40:08.821559 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:08.821447 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d-client-ca-bundle\") pod \"metrics-server-c74454f78-xph4m\" (UID: \"df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d\") " pod="openshift-monitoring/metrics-server-c74454f78-xph4m" Oct 30 21:40:08.821559 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:08.821475 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d-secret-metrics-server-client-certs\") pod \"metrics-server-c74454f78-xph4m\" (UID: \"df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d\") " pod="openshift-monitoring/metrics-server-c74454f78-xph4m" Oct 30 21:40:08.843777 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:08.843753 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-qv74l"] Oct 30 21:40:08.845798 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:08.845771 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:08.846484 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:08.846467 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-qv74l" Oct 30 21:40:08.846556 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:08.846516 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvrrj\" (UniqueName: \"kubernetes.io/projected/df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d-kube-api-access-bvrrj\") pod \"metrics-server-c74454f78-xph4m\" (UID: \"df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d\") " pod="openshift-monitoring/metrics-server-c74454f78-xph4m" Oct 30 21:40:08.872088 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:08.872061 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-qv74l"] Oct 30 21:40:08.872271 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:08.872240 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 30 21:40:08.872271 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:08.872252 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-rdbs4\"" Oct 30 21:40:08.895320 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:08.895294 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-c74454f78-xph4m" Oct 30 21:40:08.918570 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:08.918543 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9874c\" (UniqueName: \"kubernetes.io/projected/a2e2c1af-8ab2-49c6-91ac-4712b88e0571-kube-api-access-9874c\") pod \"monitoring-plugin-77f4f76594-qv74l\" (UID: \"a2e2c1af-8ab2-49c6-91ac-4712b88e0571\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-qv74l" Oct 30 21:40:08.919237 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:08.919201 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a2e2c1af-8ab2-49c6-91ac-4712b88e0571-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-qv74l\" (UID: \"a2e2c1af-8ab2-49c6-91ac-4712b88e0571\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-qv74l" Oct 30 21:40:09.020652 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:09.020614 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a2e2c1af-8ab2-49c6-91ac-4712b88e0571-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-qv74l\" (UID: \"a2e2c1af-8ab2-49c6-91ac-4712b88e0571\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-qv74l" Oct 30 21:40:09.020794 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:09.020682 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9874c\" (UniqueName: \"kubernetes.io/projected/a2e2c1af-8ab2-49c6-91ac-4712b88e0571-kube-api-access-9874c\") pod \"monitoring-plugin-77f4f76594-qv74l\" (UID: \"a2e2c1af-8ab2-49c6-91ac-4712b88e0571\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-qv74l" Oct 30 21:40:09.023100 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:09.023077 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a2e2c1af-8ab2-49c6-91ac-4712b88e0571-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-qv74l\" (UID: \"a2e2c1af-8ab2-49c6-91ac-4712b88e0571\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-qv74l" Oct 30 21:40:09.036102 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:09.036078 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9874c\" (UniqueName: \"kubernetes.io/projected/a2e2c1af-8ab2-49c6-91ac-4712b88e0571-kube-api-access-9874c\") pod \"monitoring-plugin-77f4f76594-qv74l\" (UID: \"a2e2c1af-8ab2-49c6-91ac-4712b88e0571\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-qv74l" Oct 30 21:40:09.158614 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:09.158583 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-qv74l" Oct 30 21:40:09.177074 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:09.177048 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-c74454f78-xph4m"] Oct 30 21:40:09.179885 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:40:09.179860 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf5bcaa5_7723_4ea4_bbc6_dbf65d3ecb2d.slice/crio-4492a559bb344a2b6beea07fb7d5ad30902a21e2073a035fb8b4bfe906595220 WatchSource:0}: Error finding container 4492a559bb344a2b6beea07fb7d5ad30902a21e2073a035fb8b4bfe906595220: Status 404 returned error can't find the container with id 4492a559bb344a2b6beea07fb7d5ad30902a21e2073a035fb8b4bfe906595220 Oct 30 21:40:09.230905 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:09.230858 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:09.441482 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:09.441377 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-qv74l"] Oct 30 21:40:09.444800 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:40:09.444771 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2e2c1af_8ab2_49c6_91ac_4712b88e0571.slice/crio-6c029da5dda8efd8fc8903592caf8da2944d6e2a3f6d408f726182ce379d96f4 WatchSource:0}: Error finding container 6c029da5dda8efd8fc8903592caf8da2944d6e2a3f6d408f726182ce379d96f4: Status 404 returned error can't find the container with id 6c029da5dda8efd8fc8903592caf8da2944d6e2a3f6d408f726182ce379d96f4 Oct 30 21:40:09.718190 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:09.718170 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-v9jlm" Oct 30 21:40:09.780324 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:09.780266 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-qv74l" event={"ID":"a2e2c1af-8ab2-49c6-91ac-4712b88e0571","Type":"ContainerStarted","Data":"6c029da5dda8efd8fc8903592caf8da2944d6e2a3f6d408f726182ce379d96f4"} Oct 30 21:40:09.781839 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:09.781808 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-c74454f78-xph4m" event={"ID":"df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d","Type":"ContainerStarted","Data":"4492a559bb344a2b6beea07fb7d5ad30902a21e2073a035fb8b4bfe906595220"} Oct 30 21:40:09.793466 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:09.793437 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6fd8fdc4c5-xlslr" event={"ID":"90136025-abf7-4dc6-ac4b-c3050fc417f6","Type":"ContainerStarted","Data":"c4c101bef648569161f3547c46986d276145327d75f55d951186eff3f935f9f1"} Oct 30 21:40:09.793592 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:09.793507 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6fd8fdc4c5-xlslr" event={"ID":"90136025-abf7-4dc6-ac4b-c3050fc417f6","Type":"ContainerStarted","Data":"63a41b12d14292093f5928477da4b4cf8f3753e0f67981b77fb2ada9395732f4"} Oct 30 21:40:09.793592 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:09.793554 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6fd8fdc4c5-xlslr" Oct 30 21:40:09.824537 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:09.824430 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6fd8fdc4c5-xlslr" podStartSLOduration=1.128792334 podStartE2EDuration="3.824408287s" totalImagesPullingTime="2.695615953s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:06 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:09.821945165 +0000 UTC m=+48.029524120" watchObservedRunningTime="2026-10-30 21:40:09.824408287 +0000 UTC m=+48.031987235" Oct 30 21:40:10.045580 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:10.045477 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:10.264958 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:10.264918 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:10.538711 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:10.538676 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 21:40:10.541767 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:10.541731 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:10.555070 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:10.555009 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 21:40:10.556408 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:10.555613 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 21:40:10.556408 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:10.555646 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 21:40:10.556408 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:10.555778 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 21:40:10.556408 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:10.555825 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 21:40:10.556408 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:10.555911 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 21:40:10.556408 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:10.555946 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 21:40:10.556408 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:10.556208 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 21:40:10.556408 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:10.556289 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-6slhptju3coep\"" Oct 30 21:40:10.556408 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:10.556356 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-f6sd2\"" Oct 30 21:40:10.556870 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:10.556439 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 21:40:10.557743 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:10.557722 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 21:40:10.557903 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:10.557881 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 21:40:10.558265 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:10.558238 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 21:40:10.566643 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:10.566614 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 21:40:10.567366 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:10.567347 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 21:40:10.589944 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:10.589912 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 21:40:10.642139 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:10.642099 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ldlh\" (UniqueName: \"kubernetes.io/projected/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-kube-api-access-4ldlh\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:10.642300 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:10.642145 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:10.642300 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:10.642190 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:10.642300 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:10.642211 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:10.642300 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:10.642231 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-config-out\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:10.642300 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:10.642244 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:10.642300 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:10.642267 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:10.642300 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:10.642293 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:10.642544 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:10.642330 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:10.642544 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:10.642352 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:10.642544 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:10.642397 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:10.642544 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:10.642418 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:10.642544 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:10.642446 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-config\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:10.642544 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:10.642469 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:10.642544 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:10.642484 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-web-config\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:10.642544 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:10.642504 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:10.642544 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:10.642536 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:10.642845 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:10.642553 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:10.642845 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:10.642568 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:10.642845 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:10.642606 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:10.676766 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:10.676734 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:10.743503 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:10.743478 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:10.743602 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:10.743526 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:10.743602 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:10.743560 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ldlh\" (UniqueName: \"kubernetes.io/projected/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-kube-api-access-4ldlh\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:10.743602 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:10.743585 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:10.743730 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:10.743620 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:10.743730 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:10.743646 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:10.743730 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:10.743671 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-config-out\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:10.743730 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:10.743701 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:10.743730 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:10.743724 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:10.743890 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:10.743744 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:10.743890 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:10.743780 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:10.743890 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:10.743809 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:10.743890 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:10.743843 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:10.743890 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:10.743872 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:10.744063 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:10.743898 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-config\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:10.744063 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:10.743931 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:10.744063 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:10.743950 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-web-config\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:10.744063 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:10.743984 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:10.744063 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:10.744038 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:10.744233 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:10.744068 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:10.744666 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:10.744643 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:10.744826 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:10.744703 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:10.746424 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:10.746367 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:10.746664 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:10.746424 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:10.746789 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:10.746617 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:10.748086 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:10.747776 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:10.748086 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:10.747881 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:10.748214 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:10.748129 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:10.748517 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:10.748492 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:10.749639 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:10.749142 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-config-out\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:10.749639 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:10.749587 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:10.751571 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:10.751160 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:10.751571 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:10.751541 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:10.752251 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:10.752092 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:10.752251 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:10.752204 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:10.752251 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:10.752234 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:10.753193 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:10.753169 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:10.754362 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:10.754340 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-web-config\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:10.755950 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:10.755718 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-config\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:10.777234 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:10.777212 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ldlh\" (UniqueName: \"kubernetes.io/projected/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-kube-api-access-4ldlh\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:10.803175 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:10.803149 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6fd8fdc4c5-xlslr" event={"ID":"90136025-abf7-4dc6-ac4b-c3050fc417f6","Type":"ContainerStarted","Data":"c615466039bc37a092df00550a9ec2ec2c5a376545cfc5f6ebde8412295771c7"} Oct 30 21:40:10.874629 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:10.874593 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:11.194714 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:11.194674 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 21:40:11.270747 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:11.270706 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:11.521184 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:11.521091 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:11.708446 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:11.708415 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:11.807804 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:11.807729 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-qv74l" event={"ID":"a2e2c1af-8ab2-49c6-91ac-4712b88e0571","Type":"ContainerStarted","Data":"57b68877df72c6714c5deace82b58f7574a8c45ded335758f5467b91a1c97e1e"} Oct 30 21:40:11.808149 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:11.808117 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-qv74l" Oct 30 21:40:11.809831 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:11.809800 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-c74454f78-xph4m" event={"ID":"df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d","Type":"ContainerStarted","Data":"c23ce4b9abfaee80d5fcf4fa7599d5bff47214c53f9595d17bed9ac55e3680d5"} Oct 30 21:40:11.812028 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:11.812006 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b","Type":"ContainerStarted","Data":"7421daa4284e1388825935e1de921df314559c1141dd3cb2d1b6c2954e7d62c7"} Oct 30 21:40:11.814176 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:11.814154 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-qv74l" Oct 30 21:40:11.848428 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:11.847855 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-qv74l" podStartSLOduration=2.350532645 podStartE2EDuration="3.847840422s" totalImagesPullingTime="1.497307777s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:11.847119313 +0000 UTC m=+50.054698271" watchObservedRunningTime="2026-10-30 21:40:11.847840422 +0000 UTC m=+50.055419377" Oct 30 21:40:11.901409 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:11.901333 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-c74454f78-xph4m" podStartSLOduration=2.375997058 podStartE2EDuration="3.901315741s" totalImagesPullingTime="1.525318683s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:11.899808733 +0000 UTC m=+50.107387692" watchObservedRunningTime="2026-10-30 21:40:11.901315741 +0000 UTC m=+50.108894703" Oct 30 21:40:12.816242 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:12.816211 2574 generic.go:356] "Generic (PLEG): container finished" podID="1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b" containerID="c9a2c4c7e354ff0bfe92d27ad47068f8f00d3163ec1f08a88a5b49ddd4cad80e" exitCode=0 Oct 30 21:40:12.816683 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:12.816292 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b","Type":"ContainerDied","Data":"c9a2c4c7e354ff0bfe92d27ad47068f8f00d3163ec1f08a88a5b49ddd4cad80e"} Oct 30 21:40:12.911467 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:12.911427 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:13.085560 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:13.085486 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:13.486815 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:13.486777 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:13.635798 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:13.635768 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:14.867727 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:14.867689 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:15.028498 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:15.028428 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:15.342739 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:15.342634 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:15.575415 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:15.575288 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:16.838966 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:16.838944 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6fd8fdc4c5-xlslr" Oct 30 21:40:16.843078 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:16.843055 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b","Type":"ContainerStarted","Data":"dd38eb45d35ddcf72610143f96e71fc7ee928e71fd3f8eeda09afc20bcc49e26"} Oct 30 21:40:16.843193 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:16.843155 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b","Type":"ContainerStarted","Data":"fd0dbf8a054c508e3dbab8f958d4fa7655b02969db228af9671e7c1cc9f9a883"} Oct 30 21:40:16.843257 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:16.843200 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b","Type":"ContainerStarted","Data":"d894623a6390af62bd4043a10961f6e0c09665c6d5cf9e5694c540fca70213f4"} Oct 30 21:40:16.843257 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:16.843240 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b","Type":"ContainerStarted","Data":"be222d32f0e90f6c8195bc3d07590a9be3e443b65fa6ff502ec81da31bacdf44"} Oct 30 21:40:16.843347 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:16.843278 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b","Type":"ContainerStarted","Data":"4b2bd085af3f718bc596d63c5ad8ded90ba183b796157a3432e67072c75467f7"} Oct 30 21:40:17.272922 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:17.272891 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:17.438101 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:17.438066 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:17.849880 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:17.849849 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b","Type":"ContainerStarted","Data":"a557f1280585b29505458f1f9fd878dceae79678a4f48ed21efc6d4c08bb82d8"} Oct 30 21:40:20.875028 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:20.874985 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:22.741285 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:22.741257 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:40:22.777468 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:22.777411 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=8.129264105 podStartE2EDuration="12.777376589s" totalImagesPullingTime="4.631375233s" totalInitContainerRuntime="16.737251ms" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:10 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:17.900304605 +0000 UTC m=+56.107883566" watchObservedRunningTime="2026-10-30 21:40:22.777376589 +0000 UTC m=+60.984955546" Oct 30 21:40:22.947749 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:22.947716 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:23.095337 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:23.095255 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:25.210480 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:25.210448 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:25.411718 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:25.411679 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:25.588269 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:25.588187 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:26.607322 ip-10-0-137-181 kubenswrapper[2574]: E1030 21:40:26.607284 2574 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 21:40:27.788712 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:27.788681 2574 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6db5565879-ndj8n"] Oct 30 21:40:27.932828 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:27.932799 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4-metrics-certs\") pod \"network-metrics-daemon-68gsj\" (UID: \"0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4\") " pod="openshift-multus/network-metrics-daemon-68gsj" Oct 30 21:40:27.936295 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:27.936281 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 21:40:27.945604 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:27.945582 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4-metrics-certs\") pod \"network-metrics-daemon-68gsj\" (UID: \"0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4\") " pod="openshift-multus/network-metrics-daemon-68gsj" Oct 30 21:40:28.033970 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:28.033945 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17127db1-cf66-4951-ba5d-eb486ee9d3d5-original-pull-secret\") pod \"global-pull-secret-syncer-vckqz\" (UID: \"17127db1-cf66-4951-ba5d-eb486ee9d3d5\") " pod="kube-system/global-pull-secret-syncer-vckqz" Oct 30 21:40:28.037133 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:28.037107 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 21:40:28.046271 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:28.046223 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17127db1-cf66-4951-ba5d-eb486ee9d3d5-original-pull-secret\") pod \"global-pull-secret-syncer-vckqz\" (UID: \"17127db1-cf66-4951-ba5d-eb486ee9d3d5\") " pod="kube-system/global-pull-secret-syncer-vckqz" Oct 30 21:40:28.081738 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:28.081716 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-scjcc\"" Oct 30 21:40:28.088804 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:28.088787 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vckqz" Oct 30 21:40:28.119673 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:28.119655 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4mh2p\"" Oct 30 21:40:28.127056 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:28.126855 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-68gsj" Oct 30 21:40:28.135756 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:28.135735 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tpzc\" (UniqueName: \"kubernetes.io/projected/df51787b-87eb-420c-aa69-3d2762bdd386-kube-api-access-5tpzc\") pod \"network-check-target-9h6ck\" (UID: \"df51787b-87eb-420c-aa69-3d2762bdd386\") " pod="openshift-network-diagnostics/network-check-target-9h6ck" Oct 30 21:40:28.143681 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:28.142814 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 21:40:28.149058 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:28.149036 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 21:40:28.158893 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:28.158874 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tpzc\" (UniqueName: \"kubernetes.io/projected/df51787b-87eb-420c-aa69-3d2762bdd386-kube-api-access-5tpzc\") pod \"network-check-target-9h6ck\" (UID: \"df51787b-87eb-420c-aa69-3d2762bdd386\") " pod="openshift-network-diagnostics/network-check-target-9h6ck" Oct 30 21:40:28.363458 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:28.363433 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vckqz"] Oct 30 21:40:28.366719 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:40:28.366690 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17127db1_cf66_4951_ba5d_eb486ee9d3d5.slice/crio-bf1dd2eb8c2787e807a768a34f9252c5ec2658bca90693bd3e05c13dfbfa326e WatchSource:0}: Error finding container bf1dd2eb8c2787e807a768a34f9252c5ec2658bca90693bd3e05c13dfbfa326e: Status 404 returned error can't find the container with id bf1dd2eb8c2787e807a768a34f9252c5ec2658bca90693bd3e05c13dfbfa326e Oct 30 21:40:28.367183 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:28.367170 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-ggbv4\"" Oct 30 21:40:28.374238 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:28.374217 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9h6ck" Oct 30 21:40:28.410046 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:28.410026 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-68gsj"] Oct 30 21:40:28.412659 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:40:28.412633 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0823b66f_7f92_4bd2_9dc9_7d8aa6f03fe4.slice/crio-3cff9fecf383e7f625d7e0b22fc81281e46474645ca739a07af797eb983bdede WatchSource:0}: Error finding container 3cff9fecf383e7f625d7e0b22fc81281e46474645ca739a07af797eb983bdede: Status 404 returned error can't find the container with id 3cff9fecf383e7f625d7e0b22fc81281e46474645ca739a07af797eb983bdede Oct 30 21:40:28.650325 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:28.650301 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9h6ck"] Oct 30 21:40:28.652679 ip-10-0-137-181 kubenswrapper[2574]: W1030 21:40:28.652654 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf51787b_87eb_420c_aa69_3d2762bdd386.slice/crio-2b84ff3ae32d31628d244e5d712478da442c6ecbc42c20296764c12ab103b3d0 WatchSource:0}: Error finding container 2b84ff3ae32d31628d244e5d712478da442c6ecbc42c20296764c12ab103b3d0: Status 404 returned error can't find the container with id 2b84ff3ae32d31628d244e5d712478da442c6ecbc42c20296764c12ab103b3d0 Oct 30 21:40:28.889540 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:28.889506 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9h6ck" event={"ID":"df51787b-87eb-420c-aa69-3d2762bdd386","Type":"ContainerStarted","Data":"2b84ff3ae32d31628d244e5d712478da442c6ecbc42c20296764c12ab103b3d0"} Oct 30 21:40:28.890807 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:28.890787 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-68gsj" event={"ID":"0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4","Type":"ContainerStarted","Data":"3cff9fecf383e7f625d7e0b22fc81281e46474645ca739a07af797eb983bdede"} Oct 30 21:40:28.891976 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:28.891956 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vckqz" event={"ID":"17127db1-cf66-4951-ba5d-eb486ee9d3d5","Type":"ContainerStarted","Data":"bf1dd2eb8c2787e807a768a34f9252c5ec2658bca90693bd3e05c13dfbfa326e"} Oct 30 21:40:28.896159 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:28.896140 2574 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-c74454f78-xph4m" Oct 30 21:40:28.896786 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:28.896769 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-c74454f78-xph4m" Oct 30 21:40:29.737554 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:29.737362 2574 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:29.765729 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:29.764712 2574 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:29.899123 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:29.899081 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-68gsj" event={"ID":"0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4","Type":"ContainerStarted","Data":"2042177daa9a56dd7f761338d32db722d523b91bc2be6ff7f0666bd0d31b985d"} Oct 30 21:40:29.899693 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:29.899148 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-68gsj" event={"ID":"0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4","Type":"ContainerStarted","Data":"d38330226ed2af7b566cb0bad926014eb3b589ad1755b8d2c3aecd10b73e52d3"} Oct 30 21:40:29.920936 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:29.920911 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:29.921828 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:29.921776 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-68gsj" podStartSLOduration=66.830314952 podStartE2EDuration="1m7.921757961s" totalImagesPullingTime="1.091443009s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:29.91955112 +0000 UTC m=+68.127130079" watchObservedRunningTime="2026-10-30 21:40:29.921757961 +0000 UTC m=+68.129336921" Oct 30 21:40:30.155970 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:30.155936 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:30.347783 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:30.347750 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:30.499525 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:30.499445 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:32.913789 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:32.913750 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9h6ck" event={"ID":"df51787b-87eb-420c-aa69-3d2762bdd386","Type":"ContainerStarted","Data":"87d1ccc063876a6dccef3f7647082d6d68d2750812c50bfacb784072f6dbc95d"} Oct 30 21:40:32.914252 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:32.913838 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-9h6ck" Oct 30 21:40:32.933226 ip-10-0-137-181 kubenswrapper[2574]: I1030 21:40:32.933179 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-9h6ck" podStartSLOduration=67.407235696 podStartE2EDuration="1m10.93316109s" totalImagesPullingTime="3.525925394s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:32.931989594 +0000 UTC m=+71.139568553" watchObservedRunningTime="2026-10-30 21:40:32.93316109 +0000 UTC m=+71.140740049" Oct 30 21:40:33.777745 ip-10-0-137-181 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 21:40:33.786816 ip-10-0-137-181 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 21:40:33.787032 ip-10-0-137-181 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 21:40:33.787276 ip-10-0-137-181 systemd[1]: kubelet.service: Consumed 4.664s CPU time. Oct 30 21:40:33.796980 ip-10-0-137-181 systemd[1]: Starting Kubernetes Kubelet... Oct 30 21:40:33.871282 ip-10-0-137-181 kubenswrapper[6720]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:40:33.871282 ip-10-0-137-181 kubenswrapper[6720]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 21:40:33.871282 ip-10-0-137-181 kubenswrapper[6720]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:40:33.871282 ip-10-0-137-181 kubenswrapper[6720]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:40:33.873689 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873605 6720 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:40:33.873689 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873685 6720 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:40:33.873689 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873689 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:40:33.873689 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873692 6720 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:40:33.873689 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873695 6720 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:40:33.873689 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873698 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:40:33.873925 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873702 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:40:33.873925 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873704 6720 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:40:33.873925 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873707 6720 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:40:33.873925 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873710 6720 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:40:33.873925 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873712 6720 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:40:33.873925 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873715 6720 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:40:33.873925 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873717 6720 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:40:33.873925 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873720 6720 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:40:33.873925 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873722 6720 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:40:33.873925 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873725 6720 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:40:33.873925 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873728 6720 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:40:33.873925 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873730 6720 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:40:33.873925 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873734 6720 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:40:33.873925 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873739 6720 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:40:33.873925 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873742 6720 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:40:33.873925 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873750 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:40:33.873925 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873753 6720 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:40:33.873925 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873756 6720 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:40:33.873925 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873758 6720 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:40:33.874443 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873760 6720 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:40:33.874443 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873763 6720 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:40:33.874443 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873765 6720 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:40:33.874443 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873768 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:40:33.874443 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873770 6720 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:40:33.874443 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873773 6720 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:40:33.874443 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873775 6720 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:40:33.874443 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873778 6720 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:40:33.874443 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873780 6720 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:40:33.874443 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873783 6720 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:40:33.874443 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873785 6720 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:40:33.874443 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873788 6720 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:40:33.874443 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873791 6720 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:40:33.874443 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873794 6720 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:40:33.874443 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873796 6720 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:40:33.874443 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873799 6720 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:40:33.874443 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873807 6720 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:40:33.874443 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873810 6720 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:40:33.874443 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873813 6720 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:40:33.874443 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873815 6720 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:40:33.874958 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873818 6720 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:40:33.874958 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873822 6720 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:40:33.874958 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873826 6720 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:40:33.874958 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873829 6720 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:40:33.874958 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873831 6720 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:40:33.874958 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873834 6720 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:40:33.874958 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873836 6720 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:40:33.874958 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873839 6720 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:40:33.874958 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873841 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:40:33.874958 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873843 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:40:33.874958 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873846 6720 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:40:33.874958 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873849 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:40:33.874958 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873851 6720 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:40:33.874958 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873854 6720 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:40:33.874958 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873856 6720 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:40:33.874958 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873859 6720 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:40:33.874958 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873861 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:40:33.874958 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873864 6720 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:40:33.874958 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873868 6720 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:40:33.875691 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873871 6720 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:40:33.875691 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873874 6720 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:40:33.875691 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873877 6720 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:40:33.875691 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873880 6720 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:40:33.875691 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873883 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:40:33.875691 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873885 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:40:33.875691 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873888 6720 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:40:33.875691 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873890 6720 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:40:33.875691 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873892 6720 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:40:33.875691 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873895 6720 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:40:33.875691 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873901 6720 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:40:33.875691 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873904 6720 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:40:33.875691 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873906 6720 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:40:33.875691 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873908 6720 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:40:33.875691 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873911 6720 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:40:33.875691 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873913 6720 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:40:33.875691 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873916 6720 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:40:33.875691 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873918 6720 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:40:33.875691 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873921 6720 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:40:33.875691 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873924 6720 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:40:33.876277 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873926 6720 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:40:33.876277 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873928 6720 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:40:33.876277 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873931 6720 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:40:33.876277 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873934 6720 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:40:33.876277 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873937 6720 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:40:33.876277 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873939 6720 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:40:33.876277 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873942 6720 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:40:33.876277 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873944 6720 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:40:33.876277 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873947 6720 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:40:33.876277 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873949 6720 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:40:33.876277 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873952 6720 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:40:33.876277 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873955 6720 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:40:33.876277 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873958 6720 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:40:33.876277 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873961 6720 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:40:33.876277 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873964 6720 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:40:33.876277 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873967 6720 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:40:33.876277 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873970 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:40:33.876277 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873972 6720 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:40:33.876277 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873975 6720 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:40:33.876777 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873977 6720 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:40:33.876777 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873980 6720 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:40:33.876777 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.873983 6720 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:40:33.876777 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.874823 6720 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:40:33.876777 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.874831 6720 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:40:33.876777 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.874836 6720 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:40:33.876777 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.874840 6720 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:40:33.876777 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.874843 6720 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:40:33.876777 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.874845 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:40:33.876777 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.874848 6720 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:40:33.876777 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.874851 6720 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:40:33.876777 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.874853 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:40:33.876777 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.874856 6720 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:40:33.876777 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.874858 6720 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:40:33.876777 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.874861 6720 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:40:33.876777 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.874864 6720 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:40:33.876777 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.874867 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:40:33.876777 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.874869 6720 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:40:33.876777 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.874872 6720 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:40:33.876777 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.874874 6720 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:40:33.877308 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.874877 6720 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:40:33.877308 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.874879 6720 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:40:33.877308 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.874882 6720 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:40:33.877308 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.874884 6720 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:40:33.877308 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.874887 6720 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:40:33.877308 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.874889 6720 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:40:33.877308 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.874892 6720 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:40:33.877308 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.874895 6720 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:40:33.877308 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.874897 6720 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:40:33.877308 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.874899 6720 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:40:33.877308 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.874902 6720 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:40:33.877308 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.874906 6720 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:40:33.877308 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.874908 6720 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:40:33.877308 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.874911 6720 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:40:33.877308 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.874913 6720 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:40:33.877308 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.874915 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:40:33.877308 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.874918 6720 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:40:33.877308 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.874958 6720 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:40:33.877308 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.875247 6720 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:40:33.877801 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.875256 6720 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:40:33.877801 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.875264 6720 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:40:33.877801 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.875269 6720 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:40:33.877801 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.875274 6720 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:40:33.877801 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.875279 6720 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:40:33.877801 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.875284 6720 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:40:33.877801 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.875289 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:40:33.877801 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.875293 6720 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:40:33.877801 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.875298 6720 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:40:33.877801 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.875302 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:40:33.877801 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.875306 6720 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:40:33.877801 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.875311 6720 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:40:33.877801 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.875320 6720 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:40:33.877801 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.875324 6720 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:40:33.877801 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.875328 6720 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:40:33.877801 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.875332 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:40:33.877801 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.875337 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:40:33.877801 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.875341 6720 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:40:33.877801 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.875345 6720 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:40:33.878275 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.875350 6720 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:40:33.878275 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.875354 6720 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:40:33.878275 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.875359 6720 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:40:33.878275 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.875363 6720 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:40:33.878275 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.875367 6720 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:40:33.878275 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.875377 6720 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:40:33.878275 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.875399 6720 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:40:33.878275 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.875404 6720 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:40:33.878275 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.875409 6720 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:40:33.878275 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.875414 6720 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:40:33.878275 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.875418 6720 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:40:33.878275 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.875422 6720 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:40:33.878275 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.875470 6720 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:40:33.878275 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.875478 6720 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:40:33.878275 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.875482 6720 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:40:33.878275 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.875486 6720 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:40:33.878275 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.875491 6720 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:40:33.878275 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.875500 6720 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:40:33.878275 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.875504 6720 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:40:33.878275 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.875509 6720 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:40:33.878787 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.875513 6720 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:40:33.878787 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.875517 6720 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:40:33.878787 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.875521 6720 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:40:33.878787 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.875526 6720 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:40:33.878787 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.875530 6720 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:40:33.878787 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.875534 6720 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:40:33.878787 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.875539 6720 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:40:33.878787 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.875543 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:40:33.878787 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.875547 6720 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:40:33.878787 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.875556 6720 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:40:33.878787 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.875564 6720 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:40:33.878787 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.875569 6720 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:40:33.878787 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.875574 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:40:33.878787 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.875578 6720 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:40:33.878787 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.875583 6720 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:40:33.878787 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.875587 6720 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:40:33.878787 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.875592 6720 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:40:33.878787 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.875596 6720 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:40:33.878787 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.875600 6720 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:40:33.879248 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.875604 6720 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:40:33.879248 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.875609 6720 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:40:33.879248 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.875614 6720 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:40:33.879248 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.875635 6720 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:40:33.879248 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.875653 6720 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:40:33.879248 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.875659 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:40:33.879248 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.875664 6720 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:40:33.879248 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.875669 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:40:33.879248 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.875674 6720 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:40:33.879248 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.875679 6720 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:40:33.879248 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.875684 6720 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:40:33.879248 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.875694 6720 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:40:33.879248 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.876859 6720 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:40:33.879248 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.876868 6720 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:40:33.879248 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.876871 6720 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:40:33.879248 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.876874 6720 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:40:33.879248 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.876877 6720 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:40:33.879248 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.876879 6720 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:40:33.879248 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.876882 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:40:33.879248 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.876885 6720 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:40:33.879751 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.876887 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:40:33.879751 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.876890 6720 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:40:33.879751 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.876892 6720 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:40:33.879751 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.876895 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:40:33.879751 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.876897 6720 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:40:33.879751 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.876901 6720 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:40:33.879751 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.876903 6720 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:40:33.879751 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.876906 6720 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:40:33.879751 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.876909 6720 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:40:33.879751 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.876912 6720 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:40:33.879751 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.876914 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:40:33.879751 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.876917 6720 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:40:33.879751 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.876920 6720 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:40:33.879751 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.876922 6720 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:40:33.879751 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.876925 6720 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:40:33.879751 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.876928 6720 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:40:33.879751 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.876930 6720 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:40:33.879751 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.876932 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:40:33.879751 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.876935 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:40:33.880223 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.876937 6720 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:40:33.880223 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.876940 6720 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:40:33.880223 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.876942 6720 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:40:33.880223 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.876945 6720 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:40:33.880223 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.876949 6720 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:40:33.880223 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.876951 6720 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:40:33.880223 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.876954 6720 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:40:33.880223 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.876957 6720 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:40:33.880223 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.876959 6720 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:40:33.880223 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.876962 6720 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:40:33.880223 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.876964 6720 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:40:33.880223 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.876967 6720 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:40:33.880223 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.876969 6720 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:40:33.880223 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.876972 6720 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:40:33.880223 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.876974 6720 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:40:33.880223 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.876977 6720 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:40:33.880223 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.876979 6720 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:40:33.880223 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.876983 6720 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:40:33.880223 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.876987 6720 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:40:33.880223 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.876990 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:40:33.880730 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.876994 6720 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:40:33.880730 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.876996 6720 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:40:33.880730 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.876999 6720 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:40:33.880730 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877001 6720 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:40:33.880730 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877004 6720 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:40:33.880730 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877006 6720 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:40:33.880730 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877008 6720 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:40:33.880730 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877011 6720 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:40:33.880730 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877014 6720 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:40:33.880730 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877016 6720 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:40:33.880730 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877019 6720 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:40:33.880730 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877022 6720 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:40:33.880730 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877024 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:40:33.880730 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877027 6720 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:40:33.880730 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877029 6720 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:40:33.880730 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877032 6720 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:40:33.880730 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877035 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:40:33.880730 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877038 6720 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:40:33.880730 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877042 6720 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:40:33.880730 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877045 6720 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:40:33.881252 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877047 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:40:33.881252 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877050 6720 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:40:33.881252 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877052 6720 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:40:33.881252 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877055 6720 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:40:33.881252 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877058 6720 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:40:33.881252 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877060 6720 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:40:33.881252 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877063 6720 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:40:33.881252 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877065 6720 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:40:33.881252 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877068 6720 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:40:33.881252 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877070 6720 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:40:33.881252 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877073 6720 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:40:33.881252 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877076 6720 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:40:33.881252 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877079 6720 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:40:33.881252 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877082 6720 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:40:33.881252 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877084 6720 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:40:33.881252 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877086 6720 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:40:33.881252 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877089 6720 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:40:33.881252 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877091 6720 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:40:33.881252 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877094 6720 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:40:33.881252 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877096 6720 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:40:33.881775 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877099 6720 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:40:33.881775 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877101 6720 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:40:33.881775 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877104 6720 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:40:33.881775 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877106 6720 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:40:33.881775 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877109 6720 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:40:33.881775 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877112 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:40:33.881775 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877114 6720 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:40:33.881775 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877117 6720 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:40:33.881775 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877119 6720 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:40:33.881775 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877122 6720 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:40:33.881775 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877124 6720 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:40:33.881775 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877127 6720 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:40:33.881775 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877130 6720 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:40:33.881775 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877133 6720 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:40:33.881775 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877135 6720 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:40:33.881775 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877137 6720 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:40:33.881775 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877141 6720 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:40:33.881775 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877143 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:40:33.881775 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877146 6720 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:40:33.881775 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877240 6720 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 21:40:33.882268 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877247 6720 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 21:40:33.882268 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877258 6720 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 21:40:33.882268 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877263 6720 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 21:40:33.882268 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877268 6720 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 21:40:33.882268 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877272 6720 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 21:40:33.882268 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877277 6720 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 21:40:33.882268 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877281 6720 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 21:40:33.882268 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877284 6720 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 21:40:33.882268 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877287 6720 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 21:40:33.882268 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877290 6720 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 21:40:33.882268 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877294 6720 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 21:40:33.882268 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877297 6720 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 21:40:33.882268 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877300 6720 flags.go:64] FLAG: --cgroup-root="" Oct 30 21:40:33.882268 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877303 6720 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 21:40:33.882268 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877306 6720 flags.go:64] FLAG: --client-ca-file="" Oct 30 21:40:33.882268 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877309 6720 flags.go:64] FLAG: --cloud-provider="external" Oct 30 21:40:33.882268 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877312 6720 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 21:40:33.882268 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877316 6720 flags.go:64] FLAG: --cluster-domain="" Oct 30 21:40:33.882268 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877319 6720 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 21:40:33.882268 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877323 6720 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 21:40:33.882268 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877326 6720 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 21:40:33.882268 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877329 6720 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 21:40:33.882268 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877333 6720 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 21:40:33.882268 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877336 6720 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 21:40:33.882908 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877338 6720 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 21:40:33.882908 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877342 6720 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 21:40:33.882908 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877345 6720 flags.go:64] FLAG: --contention-profiling="false" Oct 30 21:40:33.882908 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877347 6720 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 21:40:33.882908 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877350 6720 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 21:40:33.882908 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877354 6720 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 21:40:33.882908 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877357 6720 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 21:40:33.882908 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877361 6720 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 21:40:33.882908 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877364 6720 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 21:40:33.882908 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877367 6720 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 21:40:33.882908 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877370 6720 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 21:40:33.882908 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877374 6720 flags.go:64] FLAG: --enable-server="true" Oct 30 21:40:33.882908 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877377 6720 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 21:40:33.882908 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877402 6720 flags.go:64] FLAG: --event-burst="100" Oct 30 21:40:33.882908 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877405 6720 flags.go:64] FLAG: --event-qps="50" Oct 30 21:40:33.882908 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877408 6720 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 21:40:33.882908 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877411 6720 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 21:40:33.882908 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877414 6720 flags.go:64] FLAG: --eviction-hard="" Oct 30 21:40:33.882908 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877418 6720 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 21:40:33.882908 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877421 6720 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 21:40:33.882908 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877424 6720 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 21:40:33.882908 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877427 6720 flags.go:64] FLAG: --eviction-soft="" Oct 30 21:40:33.882908 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877430 6720 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 21:40:33.882908 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877432 6720 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 21:40:33.882908 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877435 6720 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 21:40:33.883541 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877438 6720 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 21:40:33.883541 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877441 6720 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 21:40:33.883541 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877444 6720 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 21:40:33.883541 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877446 6720 flags.go:64] FLAG: --feature-gates="" Oct 30 21:40:33.883541 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877450 6720 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 21:40:33.883541 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877454 6720 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 21:40:33.883541 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877457 6720 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 21:40:33.883541 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877461 6720 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 21:40:33.883541 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877463 6720 flags.go:64] FLAG: --healthz-port="10248" Oct 30 21:40:33.883541 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877466 6720 flags.go:64] FLAG: --help="false" Oct 30 21:40:33.883541 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877469 6720 flags.go:64] FLAG: --hostname-override="ip-10-0-137-181.us-east-2.compute.internal" Oct 30 21:40:33.883541 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877472 6720 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 21:40:33.883541 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877475 6720 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 21:40:33.883541 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877477 6720 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 21:40:33.883541 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877480 6720 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 21:40:33.883541 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877483 6720 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 21:40:33.883541 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877486 6720 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 21:40:33.883541 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877489 6720 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 21:40:33.883541 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877492 6720 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 21:40:33.883541 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877495 6720 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 21:40:33.883541 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877498 6720 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 21:40:33.883541 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877501 6720 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 21:40:33.883541 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877504 6720 flags.go:64] FLAG: --kube-reserved="" Oct 30 21:40:33.883541 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877506 6720 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 21:40:33.884124 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877509 6720 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 21:40:33.884124 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877512 6720 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 21:40:33.884124 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877514 6720 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 21:40:33.884124 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877517 6720 flags.go:64] FLAG: --lock-file="" Oct 30 21:40:33.884124 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877519 6720 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 21:40:33.884124 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877522 6720 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 21:40:33.884124 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877525 6720 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 21:40:33.884124 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877530 6720 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 21:40:33.884124 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877533 6720 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 21:40:33.884124 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877536 6720 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 21:40:33.884124 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877538 6720 flags.go:64] FLAG: --logging-format="text" Oct 30 21:40:33.884124 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877541 6720 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 21:40:33.884124 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877544 6720 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 21:40:33.884124 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877547 6720 flags.go:64] FLAG: --manifest-url="" Oct 30 21:40:33.884124 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877550 6720 flags.go:64] FLAG: --manifest-url-header="" Oct 30 21:40:33.884124 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877554 6720 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 21:40:33.884124 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877557 6720 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 21:40:33.884124 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877560 6720 flags.go:64] FLAG: --max-pods="110" Oct 30 21:40:33.884124 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877563 6720 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 21:40:33.884124 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877566 6720 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 21:40:33.884124 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877569 6720 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 21:40:33.884124 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877571 6720 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 21:40:33.884124 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877574 6720 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 21:40:33.884124 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877577 6720 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 21:40:33.884124 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877579 6720 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 21:40:33.884752 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877586 6720 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 21:40:33.884752 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877589 6720 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 21:40:33.884752 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877592 6720 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 21:40:33.884752 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877595 6720 flags.go:64] FLAG: --pod-cidr="" Oct 30 21:40:33.884752 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877598 6720 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 21:40:33.884752 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877601 6720 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 21:40:33.884752 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877604 6720 flags.go:64] FLAG: --pods-per-core="0" Oct 30 21:40:33.884752 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877606 6720 flags.go:64] FLAG: --port="10250" Oct 30 21:40:33.884752 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877609 6720 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 21:40:33.884752 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877612 6720 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0aac217f1957b8d85" Oct 30 21:40:33.884752 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877615 6720 flags.go:64] FLAG: --qos-reserved="" Oct 30 21:40:33.884752 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877618 6720 flags.go:64] FLAG: --read-only-port="10255" Oct 30 21:40:33.884752 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877621 6720 flags.go:64] FLAG: --register-node="true" Oct 30 21:40:33.884752 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877624 6720 flags.go:64] FLAG: --register-with-taints="" Oct 30 21:40:33.884752 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877627 6720 flags.go:64] FLAG: --registry-burst="10" Oct 30 21:40:33.884752 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877630 6720 flags.go:64] FLAG: --registry-qps="5" Oct 30 21:40:33.884752 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877633 6720 flags.go:64] FLAG: --reserved-cpus="" Oct 30 21:40:33.884752 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877635 6720 flags.go:64] FLAG: --reserved-memory="" Oct 30 21:40:33.884752 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877639 6720 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 21:40:33.884752 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877642 6720 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 21:40:33.884752 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877645 6720 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 21:40:33.884752 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877648 6720 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 21:40:33.884752 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877650 6720 flags.go:64] FLAG: --runonce="false" Oct 30 21:40:33.884752 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877653 6720 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 21:40:33.884752 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877656 6720 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 21:40:33.884752 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877659 6720 flags.go:64] FLAG: --seccomp-default="false" Oct 30 21:40:33.884752 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877661 6720 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 21:40:33.885523 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877664 6720 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 21:40:33.885523 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877667 6720 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 21:40:33.885523 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877670 6720 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 21:40:33.885523 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877673 6720 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 21:40:33.885523 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877675 6720 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 21:40:33.885523 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877678 6720 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 21:40:33.885523 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877681 6720 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 21:40:33.885523 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877684 6720 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 21:40:33.885523 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877687 6720 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 21:40:33.885523 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877692 6720 flags.go:64] FLAG: --system-cgroups="" Oct 30 21:40:33.885523 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877695 6720 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 21:40:33.885523 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877700 6720 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 21:40:33.885523 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877703 6720 flags.go:64] FLAG: --tls-cert-file="" Oct 30 21:40:33.885523 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877705 6720 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 21:40:33.885523 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877709 6720 flags.go:64] FLAG: --tls-min-version="" Oct 30 21:40:33.885523 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877711 6720 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 21:40:33.885523 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877714 6720 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 21:40:33.885523 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877717 6720 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 21:40:33.885523 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877720 6720 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 21:40:33.885523 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877722 6720 flags.go:64] FLAG: --v="2" Oct 30 21:40:33.885523 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877726 6720 flags.go:64] FLAG: --version="false" Oct 30 21:40:33.885523 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877730 6720 flags.go:64] FLAG: --vmodule="" Oct 30 21:40:33.885523 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877734 6720 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 21:40:33.885523 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.877737 6720 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 21:40:33.885523 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877881 6720 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:40:33.886124 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877885 6720 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:40:33.886124 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877889 6720 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:40:33.886124 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877891 6720 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:40:33.886124 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877894 6720 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:40:33.886124 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877896 6720 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:40:33.886124 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877899 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:40:33.886124 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877902 6720 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:40:33.886124 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877904 6720 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:40:33.886124 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877907 6720 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:40:33.886124 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877911 6720 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:40:33.886124 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877913 6720 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:40:33.886124 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877916 6720 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:40:33.886124 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877919 6720 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:40:33.886124 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877921 6720 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:40:33.886124 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877924 6720 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:40:33.886124 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877926 6720 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:40:33.886124 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877931 6720 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:40:33.886124 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877936 6720 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:40:33.886124 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877939 6720 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:40:33.886760 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877941 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:40:33.886760 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877944 6720 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:40:33.886760 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877946 6720 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:40:33.886760 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877949 6720 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:40:33.886760 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877952 6720 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:40:33.886760 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877954 6720 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:40:33.886760 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877957 6720 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:40:33.886760 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877959 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:40:33.886760 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877962 6720 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:40:33.886760 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877964 6720 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:40:33.886760 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877966 6720 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:40:33.886760 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877969 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:40:33.886760 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877972 6720 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:40:33.886760 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877974 6720 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:40:33.886760 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877977 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:40:33.886760 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877979 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:40:33.886760 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877981 6720 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:40:33.886760 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877984 6720 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:40:33.886760 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877986 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:40:33.887528 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877989 6720 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:40:33.887528 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877991 6720 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:40:33.887528 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877993 6720 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:40:33.887528 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.877997 6720 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:40:33.887528 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878000 6720 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:40:33.887528 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878002 6720 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:40:33.887528 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878004 6720 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:40:33.887528 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878007 6720 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:40:33.887528 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878010 6720 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:40:33.887528 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878013 6720 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:40:33.887528 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878018 6720 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:40:33.887528 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878020 6720 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:40:33.887528 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878023 6720 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:40:33.887528 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878025 6720 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:40:33.887528 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878028 6720 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:40:33.887528 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878030 6720 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:40:33.887528 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878033 6720 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:40:33.887528 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878035 6720 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:40:33.887528 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878038 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:40:33.887528 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878040 6720 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:40:33.888263 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878043 6720 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:40:33.888263 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878045 6720 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:40:33.888263 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878047 6720 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:40:33.888263 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878050 6720 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:40:33.888263 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878052 6720 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:40:33.888263 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878055 6720 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:40:33.888263 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878057 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:40:33.888263 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878059 6720 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:40:33.888263 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878062 6720 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:40:33.888263 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878065 6720 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:40:33.888263 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878068 6720 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:40:33.888263 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878072 6720 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:40:33.888263 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878074 6720 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:40:33.888263 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878077 6720 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:40:33.888263 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878079 6720 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:40:33.888263 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878085 6720 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:40:33.888263 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878087 6720 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:40:33.888263 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878090 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:40:33.888263 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878093 6720 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:40:33.888849 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878095 6720 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:40:33.888849 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878098 6720 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:40:33.888849 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878100 6720 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:40:33.888849 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878104 6720 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:40:33.888849 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878106 6720 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:40:33.888849 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878109 6720 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:40:33.888849 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878112 6720 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:40:33.888849 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878114 6720 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:40:33.888849 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878117 6720 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:40:33.888849 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878120 6720 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:40:33.888849 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878122 6720 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:40:33.888849 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878124 6720 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:40:33.888849 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878127 6720 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:40:33.888849 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878129 6720 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:40:33.888849 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878132 6720 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:40:33.888849 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878134 6720 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:40:33.888849 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878137 6720 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:40:33.888849 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878139 6720 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:40:33.888849 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878142 6720 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:40:33.889307 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878144 6720 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:40:33.889307 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878147 6720 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:40:33.889307 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878149 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:40:33.889307 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878152 6720 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:40:33.889307 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878154 6720 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:40:33.889307 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878157 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:40:33.889307 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878159 6720 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:40:33.889307 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878162 6720 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:40:33.889307 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878164 6720 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:40:33.889307 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.878190 6720 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:40:33.889307 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878230 6720 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:40:33.889307 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878233 6720 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:40:33.889307 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878236 6720 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:40:33.889307 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878239 6720 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:40:33.889307 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878241 6720 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:40:33.889307 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878244 6720 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:40:33.889810 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878247 6720 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:40:33.889810 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878250 6720 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:40:33.889810 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878253 6720 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:40:33.889810 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878255 6720 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:40:33.889810 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878258 6720 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:40:33.889810 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878261 6720 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:40:33.889810 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878263 6720 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:40:33.889810 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878265 6720 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:40:33.889810 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878268 6720 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:40:33.889810 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878270 6720 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:40:33.889810 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878273 6720 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:40:33.889810 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878275 6720 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:40:33.889810 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878278 6720 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:40:33.889810 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878280 6720 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:40:33.889810 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878283 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:40:33.889810 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878286 6720 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:40:33.889810 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878289 6720 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:40:33.889810 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878291 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:40:33.889810 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878294 6720 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:40:33.890376 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878296 6720 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:40:33.890376 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878299 6720 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:40:33.890376 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878301 6720 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:40:33.890376 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878303 6720 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:40:33.890376 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878306 6720 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:40:33.890376 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878309 6720 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:40:33.890376 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878311 6720 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:40:33.890376 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878315 6720 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:40:33.890376 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878317 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:40:33.890376 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878320 6720 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:40:33.890376 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878322 6720 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:40:33.890376 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878324 6720 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:40:33.890376 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878327 6720 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:40:33.890376 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878331 6720 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:40:33.890376 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878334 6720 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:40:33.890376 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878336 6720 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:40:33.890376 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878339 6720 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:40:33.890376 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878341 6720 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:40:33.890376 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878344 6720 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:40:33.890922 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878348 6720 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:40:33.890922 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878352 6720 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:40:33.890922 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878355 6720 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:40:33.890922 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878358 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:40:33.890922 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878361 6720 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:40:33.890922 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878363 6720 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:40:33.890922 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878366 6720 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:40:33.890922 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878368 6720 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:40:33.890922 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878371 6720 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:40:33.890922 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878373 6720 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:40:33.890922 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878375 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:40:33.890922 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878378 6720 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:40:33.890922 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878393 6720 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:40:33.890922 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878396 6720 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:40:33.890922 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878400 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:40:33.890922 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878403 6720 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:40:33.890922 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878405 6720 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:40:33.890922 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878408 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:40:33.890922 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878410 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:40:33.891416 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878413 6720 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:40:33.891416 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878416 6720 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:40:33.891416 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878419 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:40:33.891416 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878421 6720 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:40:33.891416 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878424 6720 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:40:33.891416 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878426 6720 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:40:33.891416 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878429 6720 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:40:33.891416 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878435 6720 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:40:33.891416 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878438 6720 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:40:33.891416 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878440 6720 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:40:33.891416 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878443 6720 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:40:33.891416 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878446 6720 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:40:33.891416 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878448 6720 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:40:33.891416 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878451 6720 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:40:33.891416 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878454 6720 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:40:33.891416 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878456 6720 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:40:33.891416 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878459 6720 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:40:33.891416 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878461 6720 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:40:33.891416 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878464 6720 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:40:33.891416 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878466 6720 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:40:33.891899 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878468 6720 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:40:33.891899 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878471 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:40:33.891899 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878473 6720 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:40:33.891899 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878476 6720 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:40:33.891899 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878478 6720 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:40:33.891899 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878480 6720 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:40:33.891899 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878483 6720 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:40:33.891899 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878485 6720 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:40:33.891899 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878488 6720 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:40:33.891899 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878490 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:40:33.891899 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878492 6720 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:40:33.891899 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878495 6720 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:40:33.891899 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878497 6720 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:40:33.891899 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878502 6720 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:40:33.891899 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878506 6720 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:40:33.891899 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878508 6720 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:40:33.891899 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878511 6720 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:40:33.891899 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878513 6720 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:40:33.891899 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878516 6720 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:40:33.892358 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878519 6720 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:40:33.892358 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878522 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:40:33.892358 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878525 6720 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:40:33.892358 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.878528 6720 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:40:33.892358 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.882340 6720 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 21:40:33.892358 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.882354 6720 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 21:40:33.892358 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.882371 6720 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 21:40:33.892358 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.882376 6720 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 21:40:33.892358 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882458 6720 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:40:33.892358 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882464 6720 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:40:33.892358 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882468 6720 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:40:33.892358 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882471 6720 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:40:33.892358 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882474 6720 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:40:33.892358 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882476 6720 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:40:33.892358 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882479 6720 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:40:33.892358 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882482 6720 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:40:33.892358 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882484 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:40:33.892358 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882487 6720 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:40:33.892358 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882489 6720 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:40:33.892826 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882492 6720 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:40:33.892826 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882494 6720 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:40:33.892826 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882498 6720 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:40:33.892826 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882502 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:40:33.892826 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882505 6720 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:40:33.892826 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882508 6720 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:40:33.892826 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882510 6720 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:40:33.892826 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882513 6720 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:40:33.892826 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882515 6720 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:40:33.892826 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882518 6720 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:40:33.892826 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882521 6720 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:40:33.892826 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882524 6720 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:40:33.892826 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882526 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:40:33.892826 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882529 6720 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:40:33.892826 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882532 6720 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:40:33.892826 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882535 6720 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:40:33.892826 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882537 6720 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:40:33.892826 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882539 6720 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:40:33.892826 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882543 6720 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:40:33.893610 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882545 6720 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:40:33.893610 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882548 6720 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:40:33.893610 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882550 6720 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:40:33.893610 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882553 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:40:33.893610 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882555 6720 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:40:33.893610 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882558 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:40:33.893610 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882561 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:40:33.893610 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882563 6720 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:40:33.893610 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882565 6720 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:40:33.893610 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882568 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:40:33.893610 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882570 6720 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:40:33.893610 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882573 6720 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:40:33.893610 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882575 6720 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:40:33.893610 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882578 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:40:33.893610 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882581 6720 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:40:33.893610 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882584 6720 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:40:33.893610 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882588 6720 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:40:33.893610 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882592 6720 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:40:33.893610 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882595 6720 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:40:33.894348 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882597 6720 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:40:33.894348 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882600 6720 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:40:33.894348 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882602 6720 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:40:33.894348 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882605 6720 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:40:33.894348 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882607 6720 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:40:33.894348 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882610 6720 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:40:33.894348 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882612 6720 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:40:33.894348 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882614 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:40:33.894348 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882617 6720 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:40:33.894348 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882619 6720 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:40:33.894348 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882622 6720 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:40:33.894348 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882624 6720 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:40:33.894348 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882626 6720 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:40:33.894348 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882629 6720 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:40:33.894348 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882632 6720 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:40:33.894348 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882634 6720 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:40:33.894348 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882637 6720 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:40:33.894348 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882639 6720 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:40:33.894348 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882642 6720 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:40:33.894348 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882644 6720 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:40:33.894869 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882646 6720 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:40:33.894869 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882649 6720 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:40:33.894869 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882651 6720 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:40:33.894869 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882653 6720 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:40:33.894869 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882656 6720 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:40:33.894869 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882658 6720 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:40:33.894869 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882661 6720 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:40:33.894869 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882663 6720 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:40:33.894869 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882665 6720 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:40:33.894869 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882668 6720 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:40:33.894869 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882671 6720 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:40:33.894869 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882674 6720 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:40:33.894869 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882676 6720 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:40:33.894869 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882678 6720 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:40:33.894869 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882681 6720 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:40:33.894869 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882683 6720 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:40:33.894869 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882686 6720 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:40:33.894869 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882688 6720 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:40:33.894869 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882691 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:40:33.895378 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882693 6720 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:40:33.895378 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882696 6720 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:40:33.895378 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882698 6720 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:40:33.895378 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882701 6720 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:40:33.895378 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882703 6720 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:40:33.895378 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882706 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:40:33.895378 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882708 6720 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:40:33.895378 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882711 6720 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:40:33.895378 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882714 6720 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:40:33.895378 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882716 6720 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:40:33.895378 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882718 6720 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:40:33.895378 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882721 6720 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:40:33.895378 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882723 6720 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:40:33.895378 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882726 6720 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:40:33.895378 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882728 6720 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:40:33.895378 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882731 6720 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:40:33.895378 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882740 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:40:33.895378 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882744 6720 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:40:33.895885 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.882772 6720 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:40:33.895885 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882901 6720 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:40:33.895885 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882906 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:40:33.895885 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882910 6720 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:40:33.895885 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882913 6720 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:40:33.895885 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882916 6720 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:40:33.895885 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882919 6720 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:40:33.895885 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882922 6720 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:40:33.895885 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882924 6720 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:40:33.895885 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882927 6720 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:40:33.895885 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882929 6720 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:40:33.895885 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882932 6720 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:40:33.895885 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882934 6720 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:40:33.895885 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882937 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:40:33.895885 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882939 6720 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:40:33.895885 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882941 6720 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:40:33.896293 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882944 6720 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:40:33.896293 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882946 6720 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:40:33.896293 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882949 6720 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:40:33.896293 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882951 6720 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:40:33.896293 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882954 6720 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:40:33.896293 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882956 6720 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:40:33.896293 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882964 6720 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:40:33.896293 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882967 6720 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:40:33.896293 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882970 6720 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:40:33.896293 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882973 6720 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:40:33.896293 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882975 6720 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:40:33.896293 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882977 6720 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:40:33.896293 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882980 6720 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:40:33.896293 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882982 6720 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:40:33.896293 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882985 6720 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:40:33.896293 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882987 6720 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:40:33.896293 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882989 6720 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:40:33.896293 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882991 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:40:33.896293 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882996 6720 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:40:33.896858 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.882999 6720 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:40:33.896858 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883002 6720 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:40:33.896858 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883005 6720 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:40:33.896858 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883008 6720 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:40:33.896858 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883010 6720 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:40:33.896858 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883013 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:40:33.896858 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883015 6720 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:40:33.896858 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883017 6720 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:40:33.896858 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883020 6720 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:40:33.896858 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883022 6720 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:40:33.896858 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883024 6720 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:40:33.896858 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883027 6720 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:40:33.896858 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883029 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:40:33.896858 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883032 6720 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:40:33.896858 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883034 6720 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:40:33.896858 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883036 6720 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:40:33.896858 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883039 6720 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:40:33.896858 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883041 6720 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:40:33.896858 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883043 6720 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:40:33.896858 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883045 6720 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:40:33.897342 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883049 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:40:33.897342 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883052 6720 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:40:33.897342 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883054 6720 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:40:33.897342 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883057 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:40:33.897342 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883059 6720 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:40:33.897342 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883063 6720 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:40:33.897342 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883066 6720 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:40:33.897342 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883069 6720 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:40:33.897342 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883072 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:40:33.897342 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883074 6720 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:40:33.897342 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883076 6720 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:40:33.897342 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883079 6720 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:40:33.897342 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883081 6720 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:40:33.897342 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883083 6720 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:40:33.897342 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883086 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:40:33.897342 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883089 6720 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:40:33.897342 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883091 6720 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:40:33.897342 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883093 6720 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:40:33.897342 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883096 6720 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:40:33.897342 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883099 6720 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:40:33.897869 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883101 6720 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:40:33.897869 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883104 6720 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:40:33.897869 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883107 6720 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:40:33.897869 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883109 6720 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:40:33.897869 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883111 6720 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:40:33.897869 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883114 6720 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:40:33.897869 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883116 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:40:33.897869 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883119 6720 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:40:33.897869 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883121 6720 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:40:33.897869 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883123 6720 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:40:33.897869 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883126 6720 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:40:33.897869 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883128 6720 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:40:33.897869 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883131 6720 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:40:33.897869 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883133 6720 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:40:33.897869 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883136 6720 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:40:33.897869 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883138 6720 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:40:33.897869 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883141 6720 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:40:33.897869 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883143 6720 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:40:33.897869 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883146 6720 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:40:33.897869 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883148 6720 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:40:33.898363 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883150 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:40:33.898363 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883153 6720 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:40:33.898363 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883155 6720 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:40:33.898363 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883157 6720 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:40:33.898363 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883160 6720 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:40:33.898363 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883163 6720 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:40:33.898363 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883165 6720 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:40:33.898363 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883168 6720 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:40:33.898363 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883170 6720 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:40:33.898363 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883172 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:40:33.898363 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883175 6720 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:40:33.898363 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883177 6720 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:40:33.898363 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.883201 6720 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:40:33.898363 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883232 6720 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:40:33.898363 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883236 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:40:33.898363 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883239 6720 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:40:33.898782 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883241 6720 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:40:33.898782 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883244 6720 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:40:33.898782 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883246 6720 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:40:33.898782 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883249 6720 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:40:33.898782 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883254 6720 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:40:33.898782 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883257 6720 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:40:33.898782 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883259 6720 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:40:33.898782 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883261 6720 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:40:33.898782 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883264 6720 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:40:33.898782 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883267 6720 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:40:33.898782 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883270 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:40:33.898782 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883272 6720 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:40:33.898782 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883275 6720 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:40:33.898782 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883277 6720 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:40:33.898782 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883279 6720 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:40:33.898782 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883282 6720 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:40:33.898782 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883284 6720 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:40:33.898782 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883287 6720 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:40:33.898782 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883289 6720 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:40:33.898782 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883292 6720 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:40:33.899299 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883294 6720 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:40:33.899299 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883297 6720 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:40:33.899299 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883300 6720 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:40:33.899299 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883302 6720 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:40:33.899299 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883304 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:40:33.899299 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883307 6720 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:40:33.899299 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883309 6720 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:40:33.899299 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883312 6720 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:40:33.899299 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883314 6720 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:40:33.899299 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883317 6720 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:40:33.899299 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883319 6720 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:40:33.899299 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883321 6720 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:40:33.899299 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883324 6720 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:40:33.899299 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883327 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:40:33.899299 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883329 6720 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:40:33.899299 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883333 6720 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:40:33.899299 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883336 6720 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:40:33.899299 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883339 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:40:33.899299 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883342 6720 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:40:33.899299 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883344 6720 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:40:33.899786 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883346 6720 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:40:33.899786 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883349 6720 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:40:33.899786 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883351 6720 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:40:33.899786 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883354 6720 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:40:33.899786 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883357 6720 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:40:33.899786 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883360 6720 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:40:33.899786 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883362 6720 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:40:33.899786 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883365 6720 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:40:33.899786 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883367 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:40:33.899786 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883369 6720 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:40:33.899786 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883372 6720 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:40:33.899786 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883374 6720 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:40:33.899786 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883377 6720 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:40:33.899786 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883395 6720 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:40:33.899786 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883400 6720 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:40:33.899786 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883402 6720 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:40:33.899786 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883405 6720 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:40:33.899786 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883407 6720 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:40:33.899786 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883410 6720 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:40:33.900250 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883413 6720 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:40:33.900250 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883416 6720 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:40:33.900250 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883418 6720 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:40:33.900250 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883421 6720 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:40:33.900250 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883423 6720 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:40:33.900250 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883426 6720 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:40:33.900250 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883428 6720 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:40:33.900250 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883431 6720 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:40:33.900250 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883435 6720 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:40:33.900250 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883439 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:40:33.900250 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883443 6720 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:40:33.900250 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883447 6720 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:40:33.900250 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883450 6720 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:40:33.900250 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883453 6720 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:40:33.900250 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883456 6720 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:40:33.900250 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883459 6720 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:40:33.900250 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883462 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:40:33.900250 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883468 6720 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:40:33.900250 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883471 6720 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:40:33.900780 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883473 6720 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:40:33.900780 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883476 6720 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:40:33.900780 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883479 6720 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:40:33.900780 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883481 6720 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:40:33.900780 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883484 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:40:33.900780 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883486 6720 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:40:33.900780 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883489 6720 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:40:33.900780 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883491 6720 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:40:33.900780 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883494 6720 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:40:33.900780 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883496 6720 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:40:33.900780 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883498 6720 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:40:33.900780 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883501 6720 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:40:33.900780 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883503 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:40:33.900780 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883506 6720 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:40:33.900780 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883508 6720 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:40:33.900780 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883511 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:40:33.900780 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883513 6720 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:40:33.900780 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883518 6720 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:40:33.900780 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883522 6720 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:40:33.900780 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883525 6720 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:40:33.901252 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883528 6720 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:40:33.901252 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883532 6720 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:40:33.901252 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883536 6720 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:40:33.901252 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883539 6720 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:40:33.901252 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:33.883541 6720 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:40:33.901252 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.883596 6720 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 21:40:33.901252 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.884513 6720 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 21:40:33.901252 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.884563 6720 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 21:40:33.901252 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.884746 6720 server.go:1014] "Starting client certificate rotation" Oct 30 21:40:33.901252 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.884904 6720 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 21:40:33.901252 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.884950 6720 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 21:34:22 +0000 UTC" deadline="2028-03-28 21:30:30.93651428 +0000 UTC" Oct 30 21:40:33.901252 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.885002 6720 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12359h49m57.051517878s" Oct 30 21:40:33.901252 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.885197 6720 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:40:33.901252 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.886257 6720 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:40:33.901252 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.887844 6720 log.go:25] "Validated CRI v1 runtime API" Oct 30 21:40:33.901252 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.889859 6720 log.go:25] "Validated CRI v1 image API" Oct 30 21:40:33.901709 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.890600 6720 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 21:40:33.901709 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.892832 6720 fs.go:135] Filesystem UUIDs: map[3a52f4fd-9681-4d1a-aea0-b982992c991a:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 9f337de4-a428-4115-ae4a-5f6d7a7095e9:/dev/nvme0n1p4] Oct 30 21:40:33.901784 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.892847 6720 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/042516063e94ac1953c1bab8314d5b5ef0ee01e12f473016848591f1489a8a69/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/042516063e94ac1953c1bab8314d5b5ef0ee01e12f473016848591f1489a8a69/userdata/shm major:0 minor:296 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/070283f63c96b42654c4445147ef147ff242174d1a6f64f36fc541812377392e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/070283f63c96b42654c4445147ef147ff242174d1a6f64f36fc541812377392e/userdata/shm major:0 minor:290 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/12db37d2844f143df7f3300e07da7aa980792b1668f8e8d66e432292f48e0257/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/12db37d2844f143df7f3300e07da7aa980792b1668f8e8d66e432292f48e0257/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1a470e35815603cf8b32f1f06dfaaa23198d564e9b3d32678e4ae65748d33fb0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1a470e35815603cf8b32f1f06dfaaa23198d564e9b3d32678e4ae65748d33fb0/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2b84ff3ae32d31628d244e5d712478da442c6ecbc42c20296764c12ab103b3d0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2b84ff3ae32d31628d244e5d712478da442c6ecbc42c20296764c12ab103b3d0/userdata/shm major:0 minor:468 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3b9f15580bfe7c628b1b43cb22aa394e3106ed959240bab23126401001e1cae5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3b9f15580bfe7c628b1b43cb22aa394e3106ed959240bab23126401001e1cae5/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3cff9fecf383e7f625d7e0b22fc81281e46474645ca739a07af797eb983bdede/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3cff9fecf383e7f625d7e0b22fc81281e46474645ca739a07af797eb983bdede/userdata/shm major:0 minor:463 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4492a559bb344a2b6beea07fb7d5ad30902a21e2073a035fb8b4bfe906595220/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4492a559bb344a2b6beea07fb7d5ad30902a21e2073a035fb8b4bfe906595220/userdata/shm major:0 minor:375 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/500e7e5992fc8ffeab24db2201da6c8c5096b9410f69f62a12141ecbab6968ac/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/500e7e5992fc8ffeab24db2201da6c8c5096b9410f69f62a12141ecbab6968ac/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5b27a017bc46424bb40d7916ec70f29c76c279cbd7086e1e482b5c3c4a06f2f9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5b27a017bc46424bb40d7916ec70f29c76c279cbd7086e1e482b5c3c4a06f2f9/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/628a1fbc58b9782013e5efba7cc6b3ac111a5174cfc6ea6086c187acca1344f4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/628a1fbc58b9782013e5efba7cc6b3ac111a5174cfc6ea6086c187acca1344f4/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6c029da5dda8efd8fc8903592caf8da2944d6e2a3f6d408f726182ce379d96f4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6c029da5dda8efd8fc8903592caf8da2944d6e2a3f6d408f726182ce379d96f4/userdata/shm major:0 minor:379 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6c704cae0f9a5285742a852fc15009b450a18501da7c1d9946d7526ad8eae067/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6c704cae0f9a5285742a852fc15009b450a18501da7c1d9946d7526ad8eae067/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7421daa4284e1388825935e1de921df314559c1141dd3cb2d1b6c2954e7d62c7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7421daa4284e1388825935e1de921df314559c1141dd3cb2d1b6c2954e7d62c7/userdata/shm major:0 minor:410 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/787d69f12c43edd684ab02c16bca74f4532a0e8ad037dc6a52df731daa0d8246/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/787d69f12c43edd684ab02c16bca74f4532a0e8ad037dc6a52df731daa0d8246/userdata/shm major:0 minor:341 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7e8c214acf99ded07a7499c3db8d9143ded44c04b3ade20919ebe4d194b3c31f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7e8c214acf99ded07a7499c3db8d9143ded44c04b3ade20919ebe4d194b3c31f/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/838254bb88a8cb6c96ecff274a46f301292ac57adff9a00097693925841d5b2d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/838254bb88a8cb6c96ecff274a46f301292ac57adff9a00097693925841d5b2d/userdata/shm major:0 minor:292 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/83ed93b8c37441e466eb5e680d93417d43d73e3a252fa206e92194540699b821/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/83ed93b8c37441e466eb5e680d93417d43d73e3a252fa206e92194540699b821/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8ced14ceba61a4a50c109a820429cb6e2ab0b84becfe9e0c9ffbc43e96b87e9d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8ced14ceba61a4a50c109a820429cb6e2ab0b84becfe9e0c9ffbc43e96b87e9d/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9e09f6b4dee3ab435d88c7e45da628d0b0d8618010b8b72516392ff86ab1b278/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9e09f6b4dee3ab435d88c7e45da628d0b0d8618010b8b72516392ff86ab1b278/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b046cd77c79cc8e11522200053c9c9c9312de2b9196a0fcbf296104cfeeefcc5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b046cd77c79cc8e11522200053c9c9c9312de2b9196a0fcbf296104cfeeefcc5/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bf1dd2eb8c2787e807a768a34f9252c5ec2658bca90693bd3e05c13dfbfa326e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bf1dd2eb8c2787e807a768a34f9252c5ec2658bca90693bd3e05c13dfbfa326e/userdata/shm major:0 minor:461 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c1d2a886d7be5a91991de6e7e298cf3d83bd72911647599177df5163ec75b12d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c1d2a886d7be5a91991de6e7e298cf3d83bd72911647599177df5163ec75b12d/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/eecd9db481580b49d59a74dcbc9a329ce9448bd64ee9ea4929683f64daf18753/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/eecd9db481580b49d59a74dcbc9a329ce9448bd64ee9ea4929683f64daf18753/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f51fed7eefea1361370e77231a9e940aeb3c0b3eed8d48dc79deca5340db157c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f51fed7eefea1361370e77231a9e940aeb3c0b3eed8d48dc79deca5340db157c/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4/volumes/kubernetes.io~projected/kube-api-access-pbcgc:{mountpoint:/var/lib/kubelet/pods/0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4/volumes/kubernetes.io~projected/kube-api-access-pbcgc major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4/volumes/kubernetes.io~secret/metrics-certs major:0 minor:456 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/17127db1-cf66-4951-ba5d-eb486ee9d3d5/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/17127db1-cf66-4951-ba5d-eb486ee9d3d5/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:460 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b/volume-subpaths/web-config/config-reloader/2 major:0 minor:412 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:412 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b/volume-subpaths/web-config/prometheus/15 major:0 minor:412 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b/volumes/kubernetes.io~empty-dir/config-out major:0 minor:405 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b/volumes/kubernetes.io~projected/kube-api-access-4ldlh:{mountpoint:/var/lib/kubelet/pods/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b/volumes/kubernetes.io~projected/kube-api-access-4ldlh major:0 minor:414 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b/volumes/kubernetes.io~projected/tls-assets major:0 minor:407 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b/volumes/kubernetes.io~secret/config major:0 minor:413 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:409 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:408 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:403 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:404 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:406 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:411 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:402 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b/volumes/kubernetes.io~secret/web-config major:0 minor:412 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/27f7c44f-0287-4608-a04a-7f108a25b121/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/27f7c44f-0287-4608-a04a-7f108a25b121/volumes/kubernetes.io~secret/agent-certs major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3018f90d-07fd-4d10-80a3-1f14248e638c/volumes/kubernetes.io~projected/kube-api-access-k7x77:{mountpoint:/var/lib/kubelet/pods/3018f90d-07fd-4d10-80a3-1f14248e638c/volumes/kubernetes.io~projected/kube-api-access-k7x77 major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3018f90d-07fd-4d10-80a3-1f14248e638c/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/3018f90d-07fd-4d10-80a3-1f14248e638c/volumes/kubernetes.io~secret/cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/451ddc8c-9938-4c6a-a210-ff9a6efd7f70/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/451ddc8c-9938-4c6a-a210-ff9a6efd7f70/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/451ddc8c-9938-4c6a-a210-ff9a6efd7f70/volumes/kubernetes.io~projected/kube-api-access-2stzq:{mountpoint:/var/lib/kubelet/pods/451ddc8c-9938-4c6a-a210-ff9a6efd7f70/volumes/kubernetes.io~projected/kube-api-access-2stzq major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/451ddc8c-9938-4c6a-a210-ff9a6efd7f70/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/451ddc8c-9938-4c6a-a210-ff9a6efd7f70/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/568ba29e-ccc3-4a4c-9a4f-192c0b931b0d/volumes/kubernetes.io~projected/kube-api-access-5wv4r:{mountpoint:/var/lib/kubelet/pods/568ba29e-ccc3-4a4c-9a4f-192c0b931b0d/volumes/kubernetes.io~projected/kube-api-access-5wv4r major:0 minor:237 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/568ba29e-ccc3-4a4c-9a4f-192c0b931b0d/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/568ba29e-ccc3-4a4c-9a4f-192c0b931b0d/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:235 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/568ba29e-ccc3-4a4c-9a4f-192c0b931b0d/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/568ba29e-ccc3-4a4c-9a4f-192c0b931b0d/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:236 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/64995343-c3c1-4915-953a-d1e4a5252461/volumes/kubernetes.io~projected/kube-api-access-c85rm:{mountpoint:/var/lib/kubelet/pods/64995343-c3c1-4915-953a-d1e4a5252461/volumes/kubernetes.io~projected/kube-api-access-c85rm major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/77670d00-a9cb-4607-bfb3-61ae91253cca/volumes/kubernetes.io~projected/kube-api-access-mgrp4:{mountpoint:/var/lib/kubelet/pods/77670d00-a9cb-4607-bfb3-61ae91253cca/volumes/kubernetes.io~projected/kube-api-access-mgrp4 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7d451a16-523d-4b43-95cc-de9eeffd8402/volumes/kubernetes.io~projected/kube-api-access-284h6:{mountpoint:/var/lib/kubelet/pods/7d451a16-523d-4b43-95cc-de9eeffd8402/volumes/kubernetes.io~projected/kube-api-access-284h6 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab/volumes/kubernetes.io~projected/kube-api-access-l725d:{mountpoint:/var/lib/kubelet/pods/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab/volumes/kubernetes.io~projected/kube-api-access-l725d major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8b0ede8c-1c7c-479c-8c04-29d849fdbf43/volumes/kubernetes.io~projected/kube-api-access-6sl5w:{mountpoint:/var/lib/kubelet/pods/8b0ede8c-1c7c-479c-8c04-29d849fdbf43/volumes/kubernetes.io~projected/kube-api-access-6sl5w major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/90136025-abf7-4dc6-ac4b-c3050fc417f6/volumes/kubernetes.io~projected/kube-api-access-6qchq:{mountpoint:/var/lib/kubelet/pods/90136025-abf7-4dc6-ac4b-c3050fc417f6/volumes/kubernetes.io~projected/kube-api-access-6qchq major:0 minor:340 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/90136025-abf7-4dc6-ac4b-c3050fc417f6/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/90136025-abf7-4dc6-ac4b-c3050fc417f6/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:335 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/90136025-abf7-4dc6-ac4b-c3050fc417f6/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/90136025-abf7-4dc6-ac4b-c3050fc417f6/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:337 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/90136025-abf7-4dc6-ac4b-c3050fc417f6/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/90136025-abf7-4dc6-ac4b-c3050fc417f6/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:336 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/90136025-abf7-4dc6-ac4b-c3050fc417f6/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/90136025-abf7-4dc6-ac4b-c3050fc417f6/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:331 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/90136025-abf7-4dc6-ac4b-c3050fc417f6/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/90136025-abf7-4dc6-ac4b-c3050fc417f6/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:339 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/90136025-abf7-4dc6-ac4b-c3050fc417f6/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/90136025-abf7-4dc6-ac4b-c3050fc417f6/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:338 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/970cd7bd-9018-4f3c-976a-af47bab1460a/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/970cd7bd-9018-4f3c-976a-af47bab1460a/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/970cd7bd-9018-4f3c-976a-af47bab1460a/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/970cd7bd-9018-4f3c-976a-af47bab1460a/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/970cd7bd-9018-4f3c-976a-af47bab1460a/volumes/kubernetes.io~projected/kube-api-access-b2kzt:{mountpoint:/var/lib/kubelet/pods/970cd7bd-9018-4f3c-976a-af47bab1460a/volumes/kubernetes.io~projected/kube-api-access-b2kzt major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/971a083e-69b8-46d1-b56a-9ccf0f4d557d/volumes/kubernetes.io~projected/kube-api-access-vfw8c:{mountpoint:/var/lib/kubelet/pods/971a083e-69b8-46d1-b56a-9ccf0f4d557d/volumes/kubernetes.io~projected/kube-api-access-vfw8c major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/971a083e-69b8-46d1-b56a-9ccf0f4d557d/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/971a083e-69b8-46d1-b56a-9ccf0f4d557d/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:277 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/971a083e-69b8-46d1-b56a-9ccf0f4d557d/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/971a083e-69b8-46d1-b56a-9ccf0f4d557d/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a2e2c1af-8ab2-49c6-91ac-4712b88e0571/volumes/kubernetes.io~projected/kube-api-access-9874c:{mountpoint:/var/lib/kubelet/pods/a2e2c1af-8ab2-49c6-91ac-4712b88e0571/volumes/kubernetes.io~projected/kube-api-access-9874c major:0 minor:378 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a2e2c1af-8ab2-49c6-91ac-4712b88e0571/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/a2e2c1af-8ab2-49c6-91ac-4712b88e0571/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:377 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b8ed5eda-cbec-4adf-93b1-0ee0f1203f30/volumes/kubernetes.io~projected/kube-api-access-gm68d:{mountpoint:/var/lib/kubelet/pods/b8ed5eda-cbec-4adf-93b1-0ee0f1203f30/volumes/kubernetes.io~projected/kube-api-access-gm68d major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b8ed5eda-cbec-4adf-93b1-0ee0f1203f30/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/b8ed5eda-cbec-4adf-93b1-0ee0f1203f30/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62/volumes/kubernetes.io~projected/kube-api-access-qpmtf:{mountpoint:/var/lib/kubelet/pods/c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62/volumes/kubernetes.io~projected/kube-api-access-qpmtf major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:285 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d0a1535a-33cb-4a99-899d-aa8b3333326f/volumes/kubernetes.io~projected/kube-api-access-sj7w5:{mountpoint:/var/lib/kubelet/pods/d0a1535a-33cb-4a99-899d-aa8b3333326f/volumes/kubernetes.io~projected/kube-api-access-sj7w5 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/df51787b-87eb-420c-aa69-3d2762bdd386/volumes/kubernetes.io~projected/kube-api-access-5tpzc:{mountpoint:/var/lib/kubelet/pods/df51787b-87eb-420c-aa69-3d2762bdd386/volumes/kubernetes.io~projected/kube-api-access-5tpzc major:0 minor:465 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d/volumes/kubernetes.io~projected/kube-api-access-bvrrj:{mountpoint:/var/lib/kubelet/pods/df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d/volumes/kubernetes.io~projected/kube-api-access-bvrrj major:0 minor:374 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:368 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:373 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:372 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e2c4958f-c7a5-4b6a-9b29-c7e2cd7d2d41/volumes/kubernetes.io~projected/kube-api-access-zlvnp:{mountpoint:/var/lib/kubelet/pods/e2c4958f-c7a5-4b6a-9b29-c7e2cd7d2d41/volumes/kubernetes.io~projected/kube-api-access-zlvnp major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e2c4958f-c7a5-4b6a-9b29-c7e2cd7d2d41/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/e2c4958f-c7a5-4b6a-9b29-c7e2cd7d2d41/volumes/kubernetes.io~secret/metrics-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ec3fe241-c753-4845-8b7a-0469c73e4c3b/volumes/kubernetes.io~projected/kube-api-access-rczmc:{mountpoint:/var/lib/kubelet/pods/ec3fe241-c753-4845-8b7a-0469c73e4c3b/volumes/kubernetes.io~projected/kube-api-access-rczmc major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ec3fe241-c753-4845-8b7a-0469c73e4c3b/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/ec3fe241-c753-4845-8b7a-0469c73e4c3b/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ec3fe241-c753-4845-8b7a-0469c73e4c3b/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/ec3fe241-c753-4845-8b7a-0469c73e4c3b/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f4b104cb-8d0c-4519-86e0-07ea0e71a4a0/volumes/kubernetes.io~projected/kube-api-access-wgq7t:{mountpoint:/var/lib/kubelet/pods/f4b104cb-8d0c-4519-86e0-07ea0e71a4a0/volumes/kubernetes.io~projected/kube-api-access-wgq7t major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f4b104cb-8d0c-4519-86e0-07ea0e71a4a0/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/f4b104cb-8d0c-4519-86e0-07ea0e71a4a0/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f4b104cb-8d0c-4519-86e0-07ea0e71a4a0/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/f4b104cb-8d0c-4519-86e0-07ea0e71a4a0/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:286 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/78c36bc91627a0db08af3093ecc9eefa00d707b46a298f477f4c99705ddf253c/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/4e7e01a6388e698cc93ed3490036924a2db68eaed8714424c6f5aa22f9087363/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/19218d3fd3c22951d8d663e3b53054d4b8409d2367958c9a5de0a15b5961c678/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/3fcc5b5d0b600bf110a9ffe88c17949f15f5e7578c8d18fd2bfd825b57019365/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/b54745d411ef7f9be0cec0668f9bbf57cb4a814f2847bce60d1bd5e3f98b147b/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/92bac107def990b7ea96da58a88474c80fec350517f2d8a7f6603f7b6233fe5e/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/cadc2dfba148427a6e415717af9d1155728d1e957fa6de4ef5f76c0b95a79d6a/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/7ab98ba3774a48d05fe2538bb611793d57c2ad51c73eec3c6e1a2c9293aca813/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/299a11b211a103e733888f3990652dfb85fdf98ddd380d416e8381aee61c6975/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/9e295198b2528aa694b529eb1579d3880d5467817bef734f8f56395e05b1d2b9/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-138:{mountpoint:/var/lib/containers/storage/overlay/50b448de70db2663f243917887e97d1522d8f93a6e7e0a455c51f871f5308fba/merged major:0 minor:138 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/a395a4e88fe57c244277e0f018b97726285cf49bff71b7527e956d62c246e34f/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/ea558de657c3729056fd60ad107073f2fa5f97235c1e5d4c48e0da25813ab6b3/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/7e8fd7a929b13c2ec316ec6ab7d493679c76351ab7b18f881616b6aacf579a55/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/3cfb94bbd222367c964bc45f429d44da71443b68573010e3a66f786826d20e80/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/10ee60e3bb9319f07d919cd0e544dfa0d7e044fdc3a261bb386145cb24543952/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-174:{mountpoint:/var/lib/containers/storage/overlay/23ec31ce7cf4b1cba650f4607dbbd2fd64ddedba5bc1fcb69a85d1fb632c1b61/merged major:0 minor:174 fsType:overlay blockSize:0} overlay_0-178:{mountpoint:/var/lib/containers/storage/overlay/229351a5dee183dd7a31204b0f37da5b12a104eeae64cb792f83192517f6ba13/merged major:0 minor:178 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/d7bd4d6a080f19eef7fa003b9a87605ac7fbd3b299b1cb47d46dfa761fab0d3b/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/ce8d8cdf8c2f73d500945327a116c5cefced23f07ccef781fe9ff6a90123efe7/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/8deacf1854d84386cfdb60271806e1c63b9439e1aad4c67aed29596c46e6f4f7/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/8b8cdc2ea142f1a1ba8c54f9bd7fbc32536f0865d1299e39752108f7940630d5/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/8d765acc253b318659a70d710b41c80f9a8cb01b36130ae3e927aa825ae20d61/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/d7b5170ee0bd3bed3fbf276cc790841449ac4ae5cd5c06b41f00a891e96236ea/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/7606bec6fd7af6eb94d2a36d6d3cc3966e651b488032c573ba94c740ce00ab4d/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-226:{mountpoint:/var/lib/containers/storage/overlay/d0b36f12ee735e59f67fbf54e10782d656c963c6c0fb09564cfd512ff805f6f6/merged major:0 minor:226 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/b4dd1d075f1308c586d6c87cb4eeb0c2d52efc448a73887c6828f9899ff7fdaa/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-240:{mountpoint:/var/lib/containers/storage/overlay/f3694a784f3f7fa3c2a43f45f1680f3f3da3ad84b0e434e87dfed493d5211bdb/merged major:0 minor:240 fsType:overlay blockSize:0} overlay_0-242:{mountpoint:/var/lib/containers/storage/overlay/435f033363c611b3d6e36458b09bb083a1cf284c08239287af519a38acd87253/merged major:0 minor:242 fsType:overlay blockSize:0} overlay_0-244:{mountpoint:/var/lib/containers/storage/overlay/35db0c5391949841cb27e38b3b5fc1372088eee25a2765298ad41fbdda3b871d/merged major:0 minor:244 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/fbafea2cd17833566519abd8b72809be68a9c5ec30830451cc973f35a0a8834f/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-255:{mountpoint:/var/lib/containers/storage/overlay/7aa66b8a6681258d1fd83ea8dc49c44331194ab480e31e87b352cbaf419aa0f2/merged major:0 minor:255 fsType:overlay blockSize:0} overlay_0-261:{mountpoint:/var/lib/containers/storage/overlay/6613023208bc6c2eb1e2aa2c5330569bd79820b3dbabdbd9a74dde03d4ed592f/merged major:0 minor:261 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/e20ec34ff1505a201aaae022b616279334343a6f68e4bb712530f4ebf3f37969/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/0818b9752cf35795434a53073c786d33a97fff1c602ffc049a61692dfa8d8fba/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/4bc97d5c1d2347fb4f8cba5ef9bdb221c42919dae0221e8df61de42df744d828/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/6e20109466ed2aa02d7e7d191273bb930252cb8b21854f5587ab325d418e9520/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/2d13e83846f07d8db05a3f57e19bede85eda2261a5d10f0b9e5b2611e5f50976/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/48e27b2b85fe8c75a8f398209d40ea1a97165cca3814cef059c8babddc42fba7/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-307:{mountpoint:/var/lib/containers/storage/overlay/25b1fb096d615a9dc564dc5f14b8852f0bad348c115755e6c124ad1348515bf8/merged major:0 minor:307 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/af6a7b9789e59c094604ca0bde2e6df15d329160d4f4a7237af66513d98f1c53/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-315:{mountpoint:/var/lib/containers/storage/overlay/b918d12fb60dafb48a9393d08f644b6b7bb3205bc1e9254337782df131676ec2/merged major:0 minor:315 fsType:overlay blockSize:0} overlay_0-320:{mountpoint:/var/lib/containers/storage/overlay/e8a5934cbb7e4bae9b8ca727b322f7489d9456b6a7fbb5419eabcfe6c2cd8524/merged major:0 minor:320 fsType:overlay blockSize:0} overlay_0-323:{mountpoint:/var/lib/containers/storage/overlay/bde3d22d5804ba626ad8c0895eb6b370ac3855309feca800f8c7b9c99492b887/merged major:0 minor:323 fsType:overlay blockSize:0} overlay_0-343:{mountpoint:/var/lib/containers/storage/overlay/7d5cb982218a6e91d7f6aea1fd14268fbce44c0a1f79d5ec127d0afbd5709715/merged major:0 minor:343 fsType:overlay blockSize:0} overlay_0-345:{mountpoint:/var/lib/containers/storage/overlay/8cc2ca8ec4170cbf3af7879017b0fa06f377950ad965ba1a904f49e5499f9801/merged major:0 minor:345 fsType:overlay blockSize:0} overlay_0-347:{mountpoint:/var/lib/containers/storage/overlay/faa116d40b3cc4b2d59ddc1d55550d577096451e8fd627a78b5204535c6c3ad2/merged major:0 minor:347 fsType:overlay blockSize:0} overlay_0-352:{mountpoint:/var/lib/containers/storage/overlay/b49d8347aa8fadd3d32dfe584adf4ea288bd8f231aefa42af47d4366ba1a4030/merged major:0 minor:352 fsType:overlay blockSize:0} overlay_0-357:{mountpoint:/var/lib/containers/storage/overlay/1f802d79522926d512af871ae40eb5cabde4f6a73c19d938909db13c0fc6390b/merged major:0 minor:357 fsType:overlay blockSize:0} overlay_0-363:{mountpoint:/var/lib/containers/storage/overlay/1c00787599b5676870060b25f29e9e7c1d3938aa85521bf00cc7ee11f7fecd9c/merged major:0 minor:363 fsType:overlay blockSize:0} overlay_0-381:{mountpoint:/var/lib/containers/storage/overlay/aee0869e76129d8efd9c08cc53c9e51f3760f18871f3711a5a9b4c74c3f0104d/merged major:0 minor:381 fsType:overlay blockSize:0} overlay_0-383:{mountpoint:/var/lib/containers/storage/overlay/ac3310874192ed1f6fb3f1f9fb6bf7380d6e7eeb0961ed74b62f857e4e140d30/merged major:0 minor:383 fsType:overlay blockSize:0} overlay_0-385:{mountpoint:/var/lib/containers/storage/overlay/ef131b0f7e2721cc0cca3a4bb8125474c8c241db3736dfea1339bc5c359106d1/merged major:0 minor:385 fsType:overlay blockSize:0} overlay_0-387:{mountpoint:/var/lib/containers/storage/overlay/5fe21388cd893f67dfdbf817bf1cfa865d1930e4376041da6ffef3eeebf400e3/merged major:0 minor:387 fsType:overlay blockSize:0} overlay_0-392:{mountpoint:/var/lib/containers/storage/overlay/eb39a2e01a9b6eb02cda5ad3e14cdb577d4e298e38b0f6109bc4e797938247f4/merged major:0 minor:392 fsType:overlay blockSize:0} overlay_0-397:{mountpoint:/var/lib/containers/storage/overlay/5b6fb423f8ebab2c7af909f182ffeed25d9377298fb7f5aa0ce513833d117cfb/merged major:0 minor:397 fsType:overlay blockSize:0} overlay_0-420:{mountpoint:/var/lib/containers/storage/overlay/e144aa69fc31c62c59b2b3234fc46f872f739c71d76f4c0e12f23bdf8540003a/merged major:0 minor:420 fsType:overlay blockSize:0} overlay_0-422:{mountpoint:/var/lib/containers/storage/overlay/7a99999bcda097ac8e6ef5d77960c94e0d88d2b94c48bf4806ea926b1de307bc/merged major:0 minor:422 fsType:overlay blockSize:0} overlay_0-428:{mountpoint:/var/lib/containers/storage/overlay/2011be6f17729699170b544840c4b191ba8d220acea3118b4281e0cf2404aebb/merged major:0 minor:428 fsType:overlay blockSize:0} overlay_0-430:{mountpoint:/var/lib/containers/storage/overlay/d20f87374715fcdb3b4634d545fcd587f376fb32b4b081daf0269720e147615b/merged major:0 minor:430 fsType:overlay blockSize:0} overlay_0-436:{mountpoint:/var/lib/containers/storage/overlay/de2b3d69c68978d09fb80dcda23281a8718172b23243687eab74b027b7a7c35f/merged major:0 minor:436 fsType:overlay blockSize:0} overlay_0-441:{mountpoint:/var/lib/containers/storage/overlay/3031525aeaceea3ae68ccf450fca0389807ec2a345655f74dc251854f1500fa4/merged major:0 minor:441 fsType:overlay blockSize:0} overlay_0-446:{mountpoint:/var/lib/containers/storage/overlay/d3e4f4e028ea5335b00cc9c7ed3cea67aa58e365b66265cda2b8ab3f4f3f8067/merged major:0 minor:446 fsType:overlay blockSize:0} overlay_0-451:{mountpoint:/var/lib/containers/storage/overlay/3c5af39608c9cd901cb6f79f17080a8acc3b8ba089552c3a7bce9ebcaedaacbb/merged major:0 minor:451 fsType:overlay blockSize:0} overlay_0-466:{mountpoint:/var/lib/containers/storage/overlay/dbc6d6b9da281fb0ee94bbe53f4b67b33abd0f08f67b81f4384eb45db1e1c23a/merged major:0 minor:466 fsType:overlay blockSize:0} overlay_0-470:{mountpoint:/var/lib/containers/storage/overlay/35875b6b282293df6865a550f4fe537abc182f6d973b8f9e0a66912e8d8dd3f0/merged major:0 minor:470 fsType:overlay blockSize:0} overlay_0-472:{mountpoint:/var/lib/containers/storage/overlay/615dfe8c2561595463644c0473a00b30e79a0c0e403f78a4824664f0d97d6673/merged major:0 minor:472 fsType:overlay blockSize:0} overlay_0-474:{mountpoint:/var/lib/containers/storage/overlay/91b71816f7db441ab0ffec30b8a1f6fbec79ca5500399dcb582113dcdabc02e5/merged major:0 minor:474 fsType:overlay blockSize:0} overlay_0-476:{mountpoint:/var/lib/containers/storage/overlay/ebbdfd6898cda80407bd24761a46e31af69e7ab479284e5040566d5e3d5157f2/merged major:0 minor:476 fsType:overlay blockSize:0} overlay_0-482:{mountpoint:/var/lib/containers/storage/overlay/a823a35db2600298f89ed46c4de5a4f3eb2e1b53f5f97725afdbfed985035c9c/merged major:0 minor:482 fsType:overlay blockSize:0} overlay_0-487:{mountpoint:/var/lib/containers/storage/overlay/4666e8565edbda1a05a9aa94e01cf481cdc949f11b190b75a7d1b83ad8535208/merged major:0 minor:487 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/b2413084748a878a93cd4c78466169b765751f5fe0fc90c171a581227d56c8fe/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/45afb49f03d89e2944502b00511dd4ae5a1858e6f6c09bf54d6d55ac9092ece7/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/77ff38e1a7562a947926d18638ff90f21b1f1d49a62ce7616a4d60145198375e/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/83dea769c2a94bf3f74e906d1f0ba20d27ee94187a919a6bb5a1f987b2079888/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/7fb8854a77ccf63051c71e4f1784ca19532f3140f05f00b2e266ab805e0623a9/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/226f7098ec9f7d5f726ad65c214b75cd811d745fa18ba33c5e03d6b741ba5ce6/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/824ee34a37a2241ec257958370d5426956e19e3974c16d79e39103880d6b4542/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/45d983272406d32141cc6d9eea509866f98db2c494315644ccb1d9b98bde7d54/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-93:{mountpoint:/var/lib/containers/storage/overlay/6b4ade3249d503ca27a124dd380171d0512b63aa7ae48b6980f9d52711ad7026/merged major:0 minor:93 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/d97f022fca3cf529ce975dd5836a5bd229334d5a7aac462256245976cefccd15/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 30 21:40:33.902091 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.901661 6720 manager.go:217] Machine: {Timestamp:2026-10-30 21:40:33.901173443 +0000 UTC m=+0.072138628 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097598 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c000c56026d053f4cbf2191645414 SystemUUID:ec2c000c-5602-6d05-3f4c-bf2191645414 BootID:5913ca45-7067-416b-bca1-1d0354b90a25 Filesystems:[{Device:/var/lib/kubelet/pods/971a083e-69b8-46d1-b56a-9ccf0f4d557d/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:277 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-93 DeviceMajor:0 DeviceMinor:93 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/451ddc8c-9938-4c6a-a210-ff9a6efd7f70/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:59 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9e09f6b4dee3ab435d88c7e45da628d0b0d8618010b8b72516392ff86ab1b278/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-242 DeviceMajor:0 DeviceMinor:242 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-307 DeviceMajor:0 DeviceMinor:307 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/970cd7bd-9018-4f3c-976a-af47bab1460a/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/27f7c44f-0287-4608-a04a-7f108a25b121/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:58 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8b0ede8c-1c7c-479c-8c04-29d849fdbf43/volumes/kubernetes.io~projected/kube-api-access-6sl5w DeviceMajor:0 DeviceMinor:67 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1a470e35815603cf8b32f1f06dfaaa23198d564e9b3d32678e4ae65748d33fb0/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/070283f63c96b42654c4445147ef147ff242174d1a6f64f36fc541812377392e/userdata/shm DeviceMajor:0 DeviceMinor:290 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:408 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:413 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b046cd77c79cc8e11522200053c9c9c9312de2b9196a0fcbf296104cfeeefcc5/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ec3fe241-c753-4845-8b7a-0469c73e4c3b/volumes/kubernetes.io~projected/kube-api-access-rczmc DeviceMajor:0 DeviceMinor:225 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4492a559bb344a2b6beea07fb7d5ad30902a21e2073a035fb8b4bfe906595220/userdata/shm DeviceMajor:0 DeviceMinor:375 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:402 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:405 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/df51787b-87eb-420c-aa69-3d2762bdd386/volumes/kubernetes.io~projected/kube-api-access-5tpzc DeviceMajor:0 DeviceMinor:465 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f51fed7eefea1361370e77231a9e940aeb3c0b3eed8d48dc79deca5340db157c/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d0a1535a-33cb-4a99-899d-aa8b3333326f/volumes/kubernetes.io~projected/kube-api-access-sj7w5 DeviceMajor:0 DeviceMinor:62 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3018f90d-07fd-4d10-80a3-1f14248e638c/volumes/kubernetes.io~projected/kube-api-access-k7x77 DeviceMajor:0 DeviceMinor:204 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6c704cae0f9a5285742a852fc15009b450a18501da7c1d9946d7526ad8eae067/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/90136025-abf7-4dc6-ac4b-c3050fc417f6/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:337 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7421daa4284e1388825935e1de921df314559c1141dd3cb2d1b6c2954e7d62c7/userdata/shm DeviceMajor:0 DeviceMinor:410 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7d451a16-523d-4b43-95cc-de9eeffd8402/volumes/kubernetes.io~projected/kube-api-access-284h6 DeviceMajor:0 DeviceMinor:66 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-138 DeviceMajor:0 DeviceMinor:138 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-174 DeviceMajor:0 DeviceMinor:174 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/eecd9db481580b49d59a74dcbc9a329ce9448bd64ee9ea4929683f64daf18753/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-422 DeviceMajor:0 DeviceMinor:422 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:412 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/17127db1-cf66-4951-ba5d-eb486ee9d3d5/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:460 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3cff9fecf383e7f625d7e0b22fc81281e46474645ca739a07af797eb983bdede/userdata/shm DeviceMajor:0 DeviceMinor:463 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/568ba29e-ccc3-4a4c-9a4f-192c0b931b0d/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:236 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-343 DeviceMajor:0 DeviceMinor:343 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d/volumes/kubernetes.io~projected/kube-api-access-bvrrj DeviceMajor:0 DeviceMinor:374 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:412 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2b84ff3ae32d31628d244e5d712478da442c6ecbc42c20296764c12ab103b3d0/userdata/shm DeviceMajor:0 DeviceMinor:468 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b8ed5eda-cbec-4adf-93b1-0ee0f1203f30/volumes/kubernetes.io~projected/kube-api-access-gm68d DeviceMajor:0 DeviceMinor:203 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c1d2a886d7be5a91991de6e7e298cf3d83bd72911647599177df5163ec75b12d/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/042516063e94ac1953c1bab8314d5b5ef0ee01e12f473016848591f1489a8a69/userdata/shm DeviceMajor:0 DeviceMinor:296 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-387 DeviceMajor:0 DeviceMinor:387 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:403 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/500e7e5992fc8ffeab24db2201da6c8c5096b9410f69f62a12141ecbab6968ac/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3b9f15580bfe7c628b1b43cb22aa394e3106ed959240bab23126401001e1cae5/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-451 DeviceMajor:0 DeviceMinor:451 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b8ed5eda-cbec-4adf-93b1-0ee0f1203f30/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:45 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f4b104cb-8d0c-4519-86e0-07ea0e71a4a0/volumes/kubernetes.io~projected/kube-api-access-wgq7t DeviceMajor:0 DeviceMinor:288 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-323 DeviceMajor:0 DeviceMinor:323 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6c029da5dda8efd8fc8903592caf8da2944d6e2a3f6d408f726182ce379d96f4/userdata/shm DeviceMajor:0 DeviceMinor:379 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-446 DeviceMajor:0 DeviceMinor:446 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-466 DeviceMajor:0 DeviceMinor:466 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-470 DeviceMajor:0 DeviceMinor:470 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/90136025-abf7-4dc6-ac4b-c3050fc417f6/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:335 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-385 DeviceMajor:0 DeviceMinor:385 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-428 DeviceMajor:0 DeviceMinor:428 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e2c4958f-c7a5-4b6a-9b29-c7e2cd7d2d41/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:201 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ec3fe241-c753-4845-8b7a-0469c73e4c3b/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:224 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ec3fe241-c753-4845-8b7a-0469c73e4c3b/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:219 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-397 DeviceMajor:0 DeviceMinor:397 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:404 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bf1dd2eb8c2787e807a768a34f9252c5ec2658bca90693bd3e05c13dfbfa326e/userdata/shm DeviceMajor:0 DeviceMinor:461 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-226 DeviceMajor:0 DeviceMinor:226 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:285 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62/volumes/kubernetes.io~projected/kube-api-access-qpmtf DeviceMajor:0 DeviceMinor:289 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/90136025-abf7-4dc6-ac4b-c3050fc417f6/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:338 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:409 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b/volumes/kubernetes.io~projected/kube-api-access-4ldlh DeviceMajor:0 DeviceMinor:414 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-430 DeviceMajor:0 DeviceMinor:430 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-482 DeviceMajor:0 DeviceMinor:482 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-244 DeviceMajor:0 DeviceMinor:244 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:287 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/787d69f12c43edd684ab02c16bca74f4532a0e8ad037dc6a52df731daa0d8246/userdata/shm DeviceMajor:0 DeviceMinor:341 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-392 DeviceMajor:0 DeviceMinor:392 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/970cd7bd-9018-4f3c-976a-af47bab1460a/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-357 DeviceMajor:0 DeviceMinor:357 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-261 DeviceMajor:0 DeviceMinor:261 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4/volumes/kubernetes.io~projected/kube-api-access-pbcgc DeviceMajor:0 DeviceMinor:55 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/83ed93b8c37441e466eb5e680d93417d43d73e3a252fa206e92194540699b821/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/90136025-abf7-4dc6-ac4b-c3050fc417f6/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:336 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:406 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-240 DeviceMajor:0 DeviceMinor:240 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-315 DeviceMajor:0 DeviceMinor:315 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/568ba29e-ccc3-4a4c-9a4f-192c0b931b0d/volumes/kubernetes.io~projected/kube-api-access-5wv4r DeviceMajor:0 DeviceMinor:237 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-352 DeviceMajor:0 DeviceMinor:352 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:372 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-487 DeviceMajor:0 DeviceMinor:487 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/971a083e-69b8-46d1-b56a-9ccf0f4d557d/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:282 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a2e2c1af-8ab2-49c6-91ac-4712b88e0571/volumes/kubernetes.io~projected/kube-api-access-9874c DeviceMajor:0 DeviceMinor:378 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/77670d00-a9cb-4607-bfb3-61ae91253cca/volumes/kubernetes.io~projected/kube-api-access-mgrp4 DeviceMajor:0 DeviceMinor:60 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-178 DeviceMajor:0 DeviceMinor:178 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/971a083e-69b8-46d1-b56a-9ccf0f4d557d/volumes/kubernetes.io~projected/kube-api-access-vfw8c DeviceMajor:0 DeviceMinor:283 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-345 DeviceMajor:0 DeviceMinor:345 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-381 DeviceMajor:0 DeviceMinor:381 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:411 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:412 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5b27a017bc46424bb40d7916ec70f29c76c279cbd7086e1e482b5c3c4a06f2f9/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f4b104cb-8d0c-4519-86e0-07ea0e71a4a0/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:286 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/90136025-abf7-4dc6-ac4b-c3050fc417f6/volumes/kubernetes.io~projected/kube-api-access-6qchq DeviceMajor:0 DeviceMinor:340 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8ced14ceba61a4a50c109a820429cb6e2ab0b84becfe9e0c9ffbc43e96b87e9d/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/628a1fbc58b9782013e5efba7cc6b3ac111a5174cfc6ea6086c187acca1344f4/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:368 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-363 DeviceMajor:0 DeviceMinor:363 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a2e2c1af-8ab2-49c6-91ac-4712b88e0571/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:377 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:456 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/e2c4958f-c7a5-4b6a-9b29-c7e2cd7d2d41/volumes/kubernetes.io~projected/kube-api-access-zlvnp DeviceMajor:0 DeviceMinor:202 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-255 DeviceMajor:0 DeviceMinor:255 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-347 DeviceMajor:0 DeviceMinor:347 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-436 DeviceMajor:0 DeviceMinor:436 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-472 DeviceMajor:0 DeviceMinor:472 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-476 DeviceMajor:0 DeviceMinor:476 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/970cd7bd-9018-4f3c-976a-af47bab1460a/volumes/kubernetes.io~projected/kube-api-access-b2kzt DeviceMajor:0 DeviceMinor:63 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3018f90d-07fd-4d10-80a3-1f14248e638c/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:197 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/568ba29e-ccc3-4a4c-9a4f-192c0b931b0d/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:235 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/90136025-abf7-4dc6-ac4b-c3050fc417f6/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:339 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/451ddc8c-9938-4c6a-a210-ff9a6efd7f70/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/451ddc8c-9938-4c6a-a210-ff9a6efd7f70/volumes/kubernetes.io~projected/kube-api-access-2stzq DeviceMajor:0 DeviceMinor:61 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/838254bb88a8cb6c96ecff274a46f301292ac57adff9a00097693925841d5b2d/userdata/shm DeviceMajor:0 DeviceMinor:292 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-441 DeviceMajor:0 DeviceMinor:441 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f4b104cb-8d0c-4519-86e0-07ea0e71a4a0/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:284 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-320 DeviceMajor:0 DeviceMinor:320 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-474 DeviceMajor:0 DeviceMinor:474 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7e8c214acf99ded07a7499c3db8d9143ded44c04b3ade20919ebe4d194b3c31f/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:373 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:407 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/64995343-c3c1-4915-953a-d1e4a5252461/volumes/kubernetes.io~projected/kube-api-access-c85rm DeviceMajor:0 DeviceMinor:65 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/12db37d2844f143df7f3300e07da7aa980792b1668f8e8d66e432292f48e0257/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-383 DeviceMajor:0 DeviceMinor:383 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-420 DeviceMajor:0 DeviceMinor:420 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab/volumes/kubernetes.io~projected/kube-api-access-l725d DeviceMajor:0 DeviceMinor:64 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/90136025-abf7-4dc6-ac4b-c3050fc417f6/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:331 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:412 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:042516063e94ac1 MacAddress:a6:12:b1:97:af:1e Speed:10000 Mtu:8901} {Name:070283f63c96b42 MacAddress:4e:d4:c1:0c:6e:ae Speed:10000 Mtu:8901} {Name:2b84ff3ae32d316 MacAddress:26:f4:cd:3f:7d:19 Speed:10000 Mtu:8901} {Name:3cff9fecf383e7f MacAddress:4e:09:73:bc:e9:41 Speed:10000 Mtu:8901} {Name:4492a559bb344a2 MacAddress:36:d5:99:da:7e:e2 Speed:10000 Mtu:8901} {Name:628a1fbc58b9782 MacAddress:1e:0a:83:df:5d:5e Speed:10000 Mtu:8901} {Name:6c029da5dda8efd MacAddress:96:e0:77:09:75:a6 Speed:10000 Mtu:8901} {Name:6c704cae0f9a528 MacAddress:5e:ff:dd:5a:72:63 Speed:10000 Mtu:8901} {Name:7421daa4284e138 MacAddress:06:3a:eb:6d:52:8f Speed:10000 Mtu:8901} {Name:787d69f12c43edd MacAddress:02:a0:8d:76:e7:c2 Speed:10000 Mtu:8901} {Name:b046cd77c79cc8e MacAddress:76:fe:6d:7c:6d:a1 Speed:10000 Mtu:8901} {Name:bf1dd2eb8c2787e MacAddress:7e:1f:a3:b3:27:50 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:ec:48:68:cd Speed:0 Mtu:9001} {Name:br-int MacAddress:ea:f9:10:f8:dd:a5 Speed:0 Mtu:8901} {Name:c1d2a886d7be5a9 MacAddress:66:ac:87:d3:ce:34 Speed:10000 Mtu:8901} {Name:eecd9db481580b4 MacAddress:6a:69:8d:f9:67:ce Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:ec:48:68:cd Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:52:b7:4c:a9:0f:f1 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:22:8f:dc:2b:68:70 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 21:40:33.902302 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.902095 6720 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 21:40:33.902302 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.902137 6720 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 21:40:33.902434 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.902405 6720 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 21:40:33.902559 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.902435 6720 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-181.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 21:40:33.902606 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.902561 6720 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 21:40:33.902606 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.902568 6720 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 21:40:33.902606 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.902578 6720 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:40:33.902606 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.902591 6720 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:40:33.902709 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.902675 6720 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 21:40:33.902865 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.902855 6720 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 21:40:33.902966 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.902957 6720 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 21:40:33.903002 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.902992 6720 kubelet.go:493] "Attempting to sync node with API server" Oct 30 21:40:33.903029 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.903005 6720 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 21:40:33.903029 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.903015 6720 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 21:40:33.903029 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.903024 6720 kubelet.go:405] "Adding apiserver pod source" Oct 30 21:40:33.903121 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.903031 6720 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 21:40:33.903894 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.903882 6720 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 21:40:33.903943 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.903895 6720 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:40:33.904349 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.904329 6720 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 21:40:33.906908 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.906889 6720 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 21:40:33.906908 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.906904 6720 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:40:33.907070 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.907028 6720 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 21:40:33.907254 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.907238 6720 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 21:40:33.907326 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.907278 6720 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 21:40:33.907420 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.907408 6720 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 21:40:33.907476 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.907429 6720 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 21:40:33.907476 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.907439 6720 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 21:40:33.907476 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.907447 6720 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 21:40:33.907476 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.907456 6720 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 21:40:33.907476 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.907465 6720 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 21:40:33.907476 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.907475 6720 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 21:40:33.907828 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.907484 6720 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 21:40:33.907828 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.907494 6720 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 21:40:33.907828 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.907503 6720 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 21:40:33.907828 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.907515 6720 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 21:40:33.907828 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.907529 6720 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 21:40:33.907828 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.907563 6720 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 21:40:33.907828 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.907572 6720 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 21:40:33.911073 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.911056 6720 server.go:1263] "Started kubelet" Oct 30 21:40:33.911169 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.911134 6720 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 21:40:33.911704 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.911650 6720 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 21:40:33.911789 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.911724 6720 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 21:40:33.911717 ip-10-0-137-181 systemd[1]: Started Kubernetes Kubelet. Oct 30 21:40:33.912211 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.912193 6720 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 21:40:33.912403 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.912368 6720 server.go:317] "Adding debug handlers to kubelet server" Oct 30 21:40:33.916903 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.916884 6720 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.916908 6720 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.916999 6720 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:34:24 +0000 UTC" deadline="2028-06-23 22:48:36.70352574 +0000 UTC" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.917019 6720 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14449h8m2.786509828s" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.917021 6720 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.917032 6720 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:33.917060 6720 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-181.us-east-2.compute.internal\" not found" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.917123 6720 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.917894 6720 factory.go:55] Registering systemd factory Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.917911 6720 factory.go:223] Registration of the systemd container factory successfully Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.918205 6720 factory.go:153] Registering CRI-O factory Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.918215 6720 factory.go:223] Registration of the crio container factory successfully Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.918256 6720 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.918269 6720 factory.go:103] Registering Raw factory Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.918280 6720 manager.go:1199] Started watching for new ooms in manager Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.918603 6720 manager.go:319] Starting recovery of all containers Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.919928 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b" volumeName="kubernetes.io/configmap/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-configmap-metrics-client-ca" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.919962 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d451a16-523d-4b43-95cc-de9eeffd8402" volumeName="kubernetes.io/configmap/7d451a16-523d-4b43-95cc-de9eeffd8402-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.919977 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62" volumeName="kubernetes.io/configmap/c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62-metrics-client-ca" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.919990 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62" volumeName="kubernetes.io/empty-dir/c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62-node-exporter-textfile" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.920004 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0a1535a-33cb-4a99-899d-aa8b3333326f" volumeName="kubernetes.io/projected/d0a1535a-33cb-4a99-899d-aa8b3333326f-kube-api-access-sj7w5" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.920017 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec3fe241-c753-4845-8b7a-0469c73e4c3b" volumeName="kubernetes.io/secret/ec3fe241-c753-4845-8b7a-0469c73e4c3b-console-serving-cert" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.920029 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b" volumeName="kubernetes.io/configmap/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-prometheus-trusted-ca-bundle" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.920041 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="568ba29e-ccc3-4a4c-9a4f-192c0b931b0d" volumeName="kubernetes.io/configmap/568ba29e-ccc3-4a4c-9a4f-192c0b931b0d-metrics-client-ca" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.920055 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="90136025-abf7-4dc6-ac4b-c3050fc417f6" volumeName="kubernetes.io/secret/90136025-abf7-4dc6-ac4b-c3050fc417f6-secret-thanos-querier-tls" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.920068 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="90136025-abf7-4dc6-ac4b-c3050fc417f6" volumeName="kubernetes.io/secret/90136025-abf7-4dc6-ac4b-c3050fc417f6-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.920081 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="90136025-abf7-4dc6-ac4b-c3050fc417f6" volumeName="kubernetes.io/secret/90136025-abf7-4dc6-ac4b-c3050fc417f6-secret-grpc-tls" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.920092 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec3fe241-c753-4845-8b7a-0469c73e4c3b" volumeName="kubernetes.io/projected/ec3fe241-c753-4845-8b7a-0469c73e4c3b-kube-api-access-rczmc" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.920104 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4b104cb-8d0c-4519-86e0-07ea0e71a4a0" volumeName="kubernetes.io/secret/f4b104cb-8d0c-4519-86e0-07ea0e71a4a0-kube-state-metrics-tls" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.920134 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b" volumeName="kubernetes.io/configmap/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-configmap-serving-certs-ca-bundle" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.920150 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b" volumeName="kubernetes.io/configmap/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-prometheus-k8s-rulefiles-1" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.920162 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b" volumeName="kubernetes.io/projected/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-tls-assets" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.920176 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b" volumeName="kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-secret-kube-rbac-proxy" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.920189 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="970cd7bd-9018-4f3c-976a-af47bab1460a" volumeName="kubernetes.io/empty-dir/970cd7bd-9018-4f3c-976a-af47bab1460a-etc-tuned" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.920201 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d" volumeName="kubernetes.io/configmap/df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d-metrics-server-audit-profiles" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.920228 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4b104cb-8d0c-4519-86e0-07ea0e71a4a0" volumeName="kubernetes.io/empty-dir/f4b104cb-8d0c-4519-86e0-07ea0e71a4a0-volume-directive-shadow" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.920241 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="451ddc8c-9938-4c6a-a210-ff9a6efd7f70" volumeName="kubernetes.io/configmap/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-ovnkube-script-lib" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.920253 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="451ddc8c-9938-4c6a-a210-ff9a6efd7f70" volumeName="kubernetes.io/configmap/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-env-overrides" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.920266 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b8ed5eda-cbec-4adf-93b1-0ee0f1203f30" volumeName="kubernetes.io/empty-dir/b8ed5eda-cbec-4adf-93b1-0ee0f1203f30-data-volume" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.920278 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62" volumeName="kubernetes.io/secret/c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.920290 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d" volumeName="kubernetes.io/projected/df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d-kube-api-access-bvrrj" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.920303 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b" volumeName="kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-secret-metrics-client-certs" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.920315 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="27f7c44f-0287-4608-a04a-7f108a25b121" volumeName="kubernetes.io/configmap/27f7c44f-0287-4608-a04a-7f108a25b121-konnectivity-ca" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.920328 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="568ba29e-ccc3-4a4c-9a4f-192c0b931b0d" volumeName="kubernetes.io/secret/568ba29e-ccc3-4a4c-9a4f-192c0b931b0d-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.920360 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="971a083e-69b8-46d1-b56a-9ccf0f4d557d" volumeName="kubernetes.io/configmap/971a083e-69b8-46d1-b56a-9ccf0f4d557d-metrics-client-ca" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.920401 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b" volumeName="kubernetes.io/projected/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-kube-api-access-4ldlh" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.920415 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b" volumeName="kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.920427 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b" volumeName="kubernetes.io/empty-dir/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-config-out" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.920441 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b" volumeName="kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-secret-grpc-tls" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.920452 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="77670d00-a9cb-4607-bfb3-61ae91253cca" volumeName="kubernetes.io/projected/77670d00-a9cb-4607-bfb3-61ae91253cca-kube-api-access-mgrp4" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.920467 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="90136025-abf7-4dc6-ac4b-c3050fc417f6" volumeName="kubernetes.io/secret/90136025-abf7-4dc6-ac4b-c3050fc417f6-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.920479 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="971a083e-69b8-46d1-b56a-9ccf0f4d557d" volumeName="kubernetes.io/projected/971a083e-69b8-46d1-b56a-9ccf0f4d557d-kube-api-access-vfw8c" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.920491 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2e2c1af-8ab2-49c6-91ac-4712b88e0571" volumeName="kubernetes.io/projected/a2e2c1af-8ab2-49c6-91ac-4712b88e0571-kube-api-access-9874c" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.920503 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62" volumeName="kubernetes.io/configmap/c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62-node-exporter-accelerators-collector-config" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.920515 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0a1535a-33cb-4a99-899d-aa8b3333326f" volumeName="kubernetes.io/configmap/d0a1535a-33cb-4a99-899d-aa8b3333326f-iptables-alerter-script" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.920528 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b" volumeName="kubernetes.io/configmap/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-prometheus-k8s-rulefiles-2" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.920541 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="451ddc8c-9938-4c6a-a210-ff9a6efd7f70" volumeName="kubernetes.io/projected/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-kube-api-access-2stzq" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.920553 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="451ddc8c-9938-4c6a-a210-ff9a6efd7f70" volumeName="kubernetes.io/secret/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.920580 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab" volumeName="kubernetes.io/configmap/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-cni-binary-copy" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.920593 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="90136025-abf7-4dc6-ac4b-c3050fc417f6" volumeName="kubernetes.io/configmap/90136025-abf7-4dc6-ac4b-c3050fc417f6-metrics-client-ca" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.920610 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df51787b-87eb-420c-aa69-3d2762bdd386" volumeName="kubernetes.io/projected/df51787b-87eb-420c-aa69-3d2762bdd386-kube-api-access-5tpzc" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.920623 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d" volumeName="kubernetes.io/empty-dir/df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d-audit-log" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.920636 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d" volumeName="kubernetes.io/secret/df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d-secret-metrics-server-client-certs" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.920648 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17127db1-cf66-4951-ba5d-eb486ee9d3d5" volumeName="kubernetes.io/secret/17127db1-cf66-4951-ba5d-eb486ee9d3d5-original-pull-secret" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.920660 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b" volumeName="kubernetes.io/configmap/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.920672 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b" volumeName="kubernetes.io/empty-dir/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-prometheus-k8s-db" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.920685 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="77670d00-a9cb-4607-bfb3-61ae91253cca" volumeName="kubernetes.io/empty-dir/77670d00-a9cb-4607-bfb3-61ae91253cca-tmp-dir" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.920697 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab" volumeName="kubernetes.io/projected/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-kube-api-access-l725d" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.920711 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="90136025-abf7-4dc6-ac4b-c3050fc417f6" volumeName="kubernetes.io/projected/90136025-abf7-4dc6-ac4b-c3050fc417f6-kube-api-access-6qchq" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.920725 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62" volumeName="kubernetes.io/projected/c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62-kube-api-access-qpmtf" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.920737 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d" volumeName="kubernetes.io/secret/df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d-client-ca-bundle" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.920749 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4" volumeName="kubernetes.io/projected/0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4-kube-api-access-pbcgc" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.920772 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b" volumeName="kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-web-config" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.920784 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="27f7c44f-0287-4608-a04a-7f108a25b121" volumeName="kubernetes.io/secret/27f7c44f-0287-4608-a04a-7f108a25b121-agent-certs" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.920797 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3018f90d-07fd-4d10-80a3-1f14248e638c" volumeName="kubernetes.io/secret/3018f90d-07fd-4d10-80a3-1f14248e638c-cert" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.920812 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d451a16-523d-4b43-95cc-de9eeffd8402" volumeName="kubernetes.io/projected/7d451a16-523d-4b43-95cc-de9eeffd8402-kube-api-access-284h6" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.920825 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b0ede8c-1c7c-479c-8c04-29d849fdbf43" volumeName="kubernetes.io/projected/8b0ede8c-1c7c-479c-8c04-29d849fdbf43-kube-api-access-6sl5w" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.920838 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2e2c1af-8ab2-49c6-91ac-4712b88e0571" volumeName="kubernetes.io/secret/a2e2c1af-8ab2-49c6-91ac-4712b88e0571-monitoring-plugin-cert" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.920850 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4b104cb-8d0c-4519-86e0-07ea0e71a4a0" volumeName="kubernetes.io/configmap/f4b104cb-8d0c-4519-86e0-07ea0e71a4a0-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.920862 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="451ddc8c-9938-4c6a-a210-ff9a6efd7f70" volumeName="kubernetes.io/configmap/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-ovnkube-config" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.920875 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b0ede8c-1c7c-479c-8c04-29d849fdbf43" volumeName="kubernetes.io/configmap/8b0ede8c-1c7c-479c-8c04-29d849fdbf43-serviceca" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.920886 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d" volumeName="kubernetes.io/configmap/df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.920898 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e2c4958f-c7a5-4b6a-9b29-c7e2cd7d2d41" volumeName="kubernetes.io/secret/e2c4958f-c7a5-4b6a-9b29-c7e2cd7d2d41-metrics-tls" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.920910 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec3fe241-c753-4845-8b7a-0469c73e4c3b" volumeName="kubernetes.io/empty-dir/ec3fe241-c753-4845-8b7a-0469c73e4c3b-tmp" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.920922 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4b104cb-8d0c-4519-86e0-07ea0e71a4a0" volumeName="kubernetes.io/projected/f4b104cb-8d0c-4519-86e0-07ea0e71a4a0-kube-api-access-wgq7t" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.920934 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="90136025-abf7-4dc6-ac4b-c3050fc417f6" volumeName="kubernetes.io/secret/90136025-abf7-4dc6-ac4b-c3050fc417f6-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.920947 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="90136025-abf7-4dc6-ac4b-c3050fc417f6" volumeName="kubernetes.io/secret/90136025-abf7-4dc6-ac4b-c3050fc417f6-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.920959 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4" volumeName="kubernetes.io/secret/0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4-metrics-certs" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.920972 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b" volumeName="kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-thanos-prometheus-http-client-file" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.920984 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b" volumeName="kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.920996 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3018f90d-07fd-4d10-80a3-1f14248e638c" volumeName="kubernetes.io/projected/3018f90d-07fd-4d10-80a3-1f14248e638c-kube-api-access-k7x77" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.921008 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab" volumeName="kubernetes.io/configmap/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-multus-daemon-config" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.921025 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="90136025-abf7-4dc6-ac4b-c3050fc417f6" volumeName="kubernetes.io/empty-dir/90136025-abf7-4dc6-ac4b-c3050fc417f6-tmp" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.921038 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b" volumeName="kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-secret-prometheus-k8s-tls" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.921050 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="970cd7bd-9018-4f3c-976a-af47bab1460a" volumeName="kubernetes.io/projected/970cd7bd-9018-4f3c-976a-af47bab1460a-kube-api-access-b2kzt" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.921062 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="971a083e-69b8-46d1-b56a-9ccf0f4d557d" volumeName="kubernetes.io/secret/971a083e-69b8-46d1-b56a-9ccf0f4d557d-openshift-state-metrics-tls" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.921075 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="971a083e-69b8-46d1-b56a-9ccf0f4d557d" volumeName="kubernetes.io/secret/971a083e-69b8-46d1-b56a-9ccf0f4d557d-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.921089 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b8ed5eda-cbec-4adf-93b1-0ee0f1203f30" volumeName="kubernetes.io/configmap/b8ed5eda-cbec-4adf-93b1-0ee0f1203f30-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.921104 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b8ed5eda-cbec-4adf-93b1-0ee0f1203f30" volumeName="kubernetes.io/projected/b8ed5eda-cbec-4adf-93b1-0ee0f1203f30-kube-api-access-gm68d" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.921116 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b8ed5eda-cbec-4adf-93b1-0ee0f1203f30" volumeName="kubernetes.io/secret/b8ed5eda-cbec-4adf-93b1-0ee0f1203f30-insights-runtime-extractor-tls" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.921130 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d" volumeName="kubernetes.io/secret/df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d-secret-metrics-server-tls" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.921142 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="64995343-c3c1-4915-953a-d1e4a5252461" volumeName="kubernetes.io/projected/64995343-c3c1-4915-953a-d1e4a5252461-kube-api-access-c85rm" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.921155 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d451a16-523d-4b43-95cc-de9eeffd8402" volumeName="kubernetes.io/configmap/7d451a16-523d-4b43-95cc-de9eeffd8402-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.921168 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62" volumeName="kubernetes.io/secret/c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62-node-exporter-tls" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.921181 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec3fe241-c753-4845-8b7a-0469c73e4c3b" volumeName="kubernetes.io/configmap/ec3fe241-c753-4845-8b7a-0469c73e4c3b-oauth-serving-cert" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.921193 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec3fe241-c753-4845-8b7a-0469c73e4c3b" volumeName="kubernetes.io/secret/ec3fe241-c753-4845-8b7a-0469c73e4c3b-console-oauth-config" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.921205 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4b104cb-8d0c-4519-86e0-07ea0e71a4a0" volumeName="kubernetes.io/configmap/f4b104cb-8d0c-4519-86e0-07ea0e71a4a0-metrics-client-ca" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.921218 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4b104cb-8d0c-4519-86e0-07ea0e71a4a0" volumeName="kubernetes.io/secret/f4b104cb-8d0c-4519-86e0-07ea0e71a4a0-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.921231 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b" volumeName="kubernetes.io/configmap/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-prometheus-k8s-rulefiles-0" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.921244 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b" volumeName="kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-config" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.921258 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="568ba29e-ccc3-4a4c-9a4f-192c0b931b0d" volumeName="kubernetes.io/projected/568ba29e-ccc3-4a4c-9a4f-192c0b931b0d-kube-api-access-5wv4r" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.921270 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="970cd7bd-9018-4f3c-976a-af47bab1460a" volumeName="kubernetes.io/empty-dir/970cd7bd-9018-4f3c-976a-af47bab1460a-tmp" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.921282 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e2c4958f-c7a5-4b6a-9b29-c7e2cd7d2d41" volumeName="kubernetes.io/configmap/e2c4958f-c7a5-4b6a-9b29-c7e2cd7d2d41-config-volume" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.921295 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e2c4958f-c7a5-4b6a-9b29-c7e2cd7d2d41" volumeName="kubernetes.io/empty-dir/e2c4958f-c7a5-4b6a-9b29-c7e2cd7d2d41-tmp-dir" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.921307 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e2c4958f-c7a5-4b6a-9b29-c7e2cd7d2d41" volumeName="kubernetes.io/projected/e2c4958f-c7a5-4b6a-9b29-c7e2cd7d2d41-kube-api-access-zlvnp" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.921318 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec3fe241-c753-4845-8b7a-0469c73e4c3b" volumeName="kubernetes.io/configmap/ec3fe241-c753-4845-8b7a-0469c73e4c3b-console-config" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.921332 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="568ba29e-ccc3-4a4c-9a4f-192c0b931b0d" volumeName="kubernetes.io/secret/568ba29e-ccc3-4a4c-9a4f-192c0b931b0d-prometheus-operator-tls" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.921344 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d451a16-523d-4b43-95cc-de9eeffd8402" volumeName="kubernetes.io/configmap/7d451a16-523d-4b43-95cc-de9eeffd8402-cni-binary-copy" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.921357 6720 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec3fe241-c753-4845-8b7a-0469c73e4c3b" volumeName="kubernetes.io/configmap/ec3fe241-c753-4845-8b7a-0469c73e4c3b-service-ca" seLinuxMountContext="" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.921369 6720 reconstruct.go:97] "Volume reconstruction finished" Oct 30 21:40:33.921526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.921377 6720 reconciler.go:29] "Reconciler: start to sync state" Oct 30 21:40:33.927217 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.924588 6720 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:40:33.927217 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.926315 6720 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:40:33.929713 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.929658 6720 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 21:40:33.931085 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.931059 6720 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:40:33.946727 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.946698 6720 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 21:40:33.948478 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.948457 6720 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 21:40:33.948576 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.948485 6720 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 21:40:33.948645 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.948587 6720 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 21:40:33.948645 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:33.948634 6720 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 21:40:33.952800 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.952779 6720 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:40:33.958310 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.958265 6720 generic.go:356] "Generic (PLEG): container finished" podID="7d451a16-523d-4b43-95cc-de9eeffd8402" containerID="0e6753e335a8c6154e619eb13b8dda7efd02f1e3d559a685c4bac597fc9ca933" exitCode=0 Oct 30 21:40:33.958310 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.958291 6720 generic.go:356] "Generic (PLEG): container finished" podID="7d451a16-523d-4b43-95cc-de9eeffd8402" containerID="b59ba665a148603ebae2d612869af6b6cb0f4d5bb3c1e5b20d3bed920d470ffa" exitCode=0 Oct 30 21:40:33.958310 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.958301 6720 generic.go:356] "Generic (PLEG): container finished" podID="7d451a16-523d-4b43-95cc-de9eeffd8402" containerID="6210176d68c45fcbf4ef9ea858cac9570fc96357d3eda865dd182425f30d23cd" exitCode=0 Oct 30 21:40:33.958310 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.958310 6720 generic.go:356] "Generic (PLEG): container finished" podID="7d451a16-523d-4b43-95cc-de9eeffd8402" containerID="acffc7d3523044b00355e9b09ea917ff7ee5d1d8defd3090810f8ef0e44f3e1d" exitCode=0 Oct 30 21:40:33.958553 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.958318 6720 generic.go:356] "Generic (PLEG): container finished" podID="7d451a16-523d-4b43-95cc-de9eeffd8402" containerID="60decec7eda6240250eed6c1ea5d5a2ce7e8ce2753f77101d17084d4056b0d73" exitCode=0 Oct 30 21:40:33.958553 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.958329 6720 generic.go:356] "Generic (PLEG): container finished" podID="7d451a16-523d-4b43-95cc-de9eeffd8402" containerID="7e67e5bbc1b09eb2355170dd2dd33ce342a445d8c026cae65701dc82a7192e31" exitCode=0 Oct 30 21:40:33.967336 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.967315 6720 generic.go:356] "Generic (PLEG): container finished" podID="1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b" containerID="c9a2c4c7e354ff0bfe92d27ad47068f8f00d3163ec1f08a88a5b49ddd4cad80e" exitCode=0 Oct 30 21:40:33.979441 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.979413 6720 generic.go:356] "Generic (PLEG): container finished" podID="952d6c48a2bc0d00a5a217ec8deb4cd1" containerID="8b498db6bcfaca8f1bd378bd2c97fcb7b9d3daf34ec40e61008e8fd1b7f7ee00" exitCode=0 Oct 30 21:40:33.985354 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:33.985332 6720 generic.go:356] "Generic (PLEG): container finished" podID="c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62" containerID="3c28e35963f034e787dd1415b9ea422cbe50c5cf9343a50b978b6f619bc1ae98" exitCode=0 Oct 30 21:40:34.032099 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.032037 6720 manager.go:324] Recovery completed Oct 30 21:40:34.049225 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:34.049197 6720 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 30 21:40:34.056137 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.056112 6720 cpu_manager.go:226] "Starting" policy="none" Oct 30 21:40:34.056137 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.056128 6720 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 21:40:34.056137 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.056143 6720 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 21:40:34.056332 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.056264 6720 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 21:40:34.056332 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.056274 6720 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 21:40:34.056332 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.056289 6720 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 21:40:34.056332 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.056294 6720 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 21:40:34.056332 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.056299 6720 policy_none.go:50] "Start" Oct 30 21:40:34.056332 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.056307 6720 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 21:40:34.056332 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.056316 6720 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 21:40:34.063474 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.063459 6720 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 21:40:34.063474 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.063473 6720 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 21:40:34.063474 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.063480 6720 policy_none.go:44] "Start" Oct 30 21:40:34.067952 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.067925 6720 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 21:40:34.071910 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.071890 6720 manager.go:341] "Starting Device Plugin manager" Oct 30 21:40:34.072398 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:34.071921 6720 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 21:40:34.072445 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.072406 6720 server.go:87] "Starting device plugin registration server" Oct 30 21:40:34.072677 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.072667 6720 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 21:40:34.072707 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.072679 6720 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 21:40:34.072800 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.072783 6720 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 21:40:34.072919 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.072882 6720 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 21:40:34.073031 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.072925 6720 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 21:40:34.073031 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.072930 6720 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 21:40:34.073031 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.072978 6720 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T21:40:34.072914528Z","UUID":"d180a5f8-3a31-4835-818b-f1ee2a0b8d99","Handler":null,"Name":"","Endpoint":""} Oct 30 21:40:34.074197 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.074177 6720 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 21:40:34.074311 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.074213 6720 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 21:40:34.074535 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:34.074516 6720 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 21:40:34.173583 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.173557 6720 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:40:34.175582 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.175556 6720 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-181.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:40:34.175674 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.175590 6720 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-181.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:40:34.175674 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.175600 6720 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-181.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:40:34.175674 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.175640 6720 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-137-181.us-east-2.compute.internal" Oct 30 21:40:34.193584 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.193567 6720 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-137-181.us-east-2.compute.internal" Oct 30 21:40:34.193625 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.193620 6720 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-137-181.us-east-2.compute.internal" Oct 30 21:40:34.249624 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.249571 6720 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-181.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-181.us-east-2.compute.internal"] Oct 30 21:40:34.250214 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.250200 6720 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-181.us-east-2.compute.internal" Oct 30 21:40:34.250274 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.250257 6720 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-181.us-east-2.compute.internal" Oct 30 21:40:34.250330 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.250187 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-181.us-east-2.compute.internal" event={"ID":"5fb540c197b86dcf7fd282ae878b96ce","Type":"ContainerStarted","Data":"5096e5426c8feab9e02ba3d6ba73bcfda77a07fd269385fe9b0e688fa35cc80b"} Oct 30 21:40:34.250330 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.250314 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-181.us-east-2.compute.internal" event={"ID":"5fb540c197b86dcf7fd282ae878b96ce","Type":"ContainerStarted","Data":"f51fed7eefea1361370e77231a9e940aeb3c0b3eed8d48dc79deca5340db157c"} Oct 30 21:40:34.250449 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.250434 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-181.us-east-2.compute.internal" event={"ID":"952d6c48a2bc0d00a5a217ec8deb4cd1","Type":"ContainerStarted","Data":"3784e02ec16c2a6804995b303ca638074df0681aa619200779f593ff3d47dcd3"} Oct 30 21:40:34.250489 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.250462 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-181.us-east-2.compute.internal" event={"ID":"952d6c48a2bc0d00a5a217ec8deb4cd1","Type":"ContainerDied","Data":"8b498db6bcfaca8f1bd378bd2c97fcb7b9d3daf34ec40e61008e8fd1b7f7ee00"} Oct 30 21:40:34.250523 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.250492 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-181.us-east-2.compute.internal" event={"ID":"952d6c48a2bc0d00a5a217ec8deb4cd1","Type":"ContainerStarted","Data":"7e8c214acf99ded07a7499c3db8d9143ded44c04b3ade20919ebe4d194b3c31f"} Oct 30 21:40:34.265510 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.265493 6720 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:40:34.265585 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:34.265529 6720 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-137-181.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-181.us-east-2.compute.internal" Oct 30 21:40:34.266835 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.266821 6720 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:40:34.266882 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:34.266854 6720 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-137-181.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-181.us-east-2.compute.internal" Oct 30 21:40:34.332209 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.332147 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/952d6c48a2bc0d00a5a217ec8deb4cd1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-181.us-east-2.compute.internal\" (UID: \"952d6c48a2bc0d00a5a217ec8deb4cd1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-181.us-east-2.compute.internal" Oct 30 21:40:34.332209 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.332179 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/952d6c48a2bc0d00a5a217ec8deb4cd1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-181.us-east-2.compute.internal\" (UID: \"952d6c48a2bc0d00a5a217ec8deb4cd1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-181.us-east-2.compute.internal" Oct 30 21:40:34.332209 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.332199 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5fb540c197b86dcf7fd282ae878b96ce-config\") pod \"kube-apiserver-proxy-ip-10-0-137-181.us-east-2.compute.internal\" (UID: \"5fb540c197b86dcf7fd282ae878b96ce\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-181.us-east-2.compute.internal" Oct 30 21:40:34.432692 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.432665 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/952d6c48a2bc0d00a5a217ec8deb4cd1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-181.us-east-2.compute.internal\" (UID: \"952d6c48a2bc0d00a5a217ec8deb4cd1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-181.us-east-2.compute.internal" Oct 30 21:40:34.432790 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.432693 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/952d6c48a2bc0d00a5a217ec8deb4cd1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-181.us-east-2.compute.internal\" (UID: \"952d6c48a2bc0d00a5a217ec8deb4cd1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-181.us-east-2.compute.internal" Oct 30 21:40:34.432790 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.432714 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/952d6c48a2bc0d00a5a217ec8deb4cd1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-181.us-east-2.compute.internal\" (UID: \"952d6c48a2bc0d00a5a217ec8deb4cd1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-181.us-east-2.compute.internal" Oct 30 21:40:34.432790 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.432732 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5fb540c197b86dcf7fd282ae878b96ce-config\") pod \"kube-apiserver-proxy-ip-10-0-137-181.us-east-2.compute.internal\" (UID: \"5fb540c197b86dcf7fd282ae878b96ce\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-181.us-east-2.compute.internal" Oct 30 21:40:34.432790 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.432766 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5fb540c197b86dcf7fd282ae878b96ce-config\") pod \"kube-apiserver-proxy-ip-10-0-137-181.us-east-2.compute.internal\" (UID: \"5fb540c197b86dcf7fd282ae878b96ce\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-181.us-east-2.compute.internal" Oct 30 21:40:34.432938 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.432806 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/952d6c48a2bc0d00a5a217ec8deb4cd1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-181.us-east-2.compute.internal\" (UID: \"952d6c48a2bc0d00a5a217ec8deb4cd1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-181.us-east-2.compute.internal" Oct 30 21:40:34.903380 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.903349 6720 apiserver.go:52] "Watching apiserver" Oct 30 21:40:34.921151 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.921127 6720 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 21:40:34.921836 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.921812 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-tj676","openshift-multus/multus-additional-cni-plugins-j8vz9","kube-system/global-pull-secret-syncer-vckqz","kube-system/konnectivity-agent-jg4wr","openshift-monitoring/metrics-server-c74454f78-xph4m","openshift-network-operator/iptables-alerter-bzhn4","openshift-monitoring/prometheus-operator-9cd89b74c-hrkzk","openshift-insights/insights-runtime-extractor-bhdkg","openshift-monitoring/node-exporter-hbg8k","openshift-multus/multus-25nw6","openshift-cluster-node-tuning-operator/tuned-vsgzd","openshift-ovn-kubernetes/ovnkube-node-75z8j","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-181.us-east-2.compute.internal","openshift-image-registry/node-ca-m7qjt","openshift-network-diagnostics/network-check-target-9h6ck","openshift-monitoring/monitoring-plugin-77f4f76594-qv74l","openshift-monitoring/openshift-state-metrics-7bd7489957-44hwx","openshift-dns/dns-default-v9jlm","openshift-monitoring/kube-state-metrics-5dc8d7d89b-4xbzf","openshift-monitoring/prometheus-k8s-0","openshift-monitoring/thanos-querier-6fd8fdc4c5-xlslr","kube-system/kube-apiserver-proxy-ip-10-0-137-181.us-east-2.compute.internal","openshift-dns/node-resolver-pnfw6","openshift-console/console-6db5565879-ndj8n","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kf9nx","openshift-multus/network-metrics-daemon-68gsj"] Oct 30 21:40:34.928191 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.928154 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-wpx2j\"" Oct 30 21:40:34.928303 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.928230 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-scjcc\"" Oct 30 21:40:34.930507 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.930489 6720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 21:40:34.930953 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.930929 6720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 21:40:34.931227 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.931210 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 21:40:34.931334 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.931317 6720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 21:40:34.931469 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.931448 6720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:40:34.931620 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.931553 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 21:40:34.931850 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.931214 6720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 21:40:34.931850 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.931845 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 21:40:34.931967 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.931789 6720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 21:40:34.931967 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.931935 6720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 21:40:34.932632 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.932618 6720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 21:40:34.933047 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.933017 6720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 21:40:34.933117 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.933070 6720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 21:40:34.933117 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.933097 6720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:40:34.933207 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.933160 6720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 21:40:34.933265 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.933205 6720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 21:40:34.933265 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.933075 6720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 21:40:34.933265 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.933245 6720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 21:40:34.933412 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.933270 6720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 21:40:34.933412 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.933274 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 21:40:34.933412 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.933307 6720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 21:40:34.933565 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.933464 6720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 21:40:34.933565 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.933480 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-6cdkd\"" Oct 30 21:40:34.933663 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.933587 6720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 21:40:34.933726 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.933663 6720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 21:40:34.933772 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.933731 6720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 21:40:34.933818 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.933801 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-h2pxp\"" Oct 30 21:40:34.933881 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.933852 6720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 21:40:34.933916 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.933886 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-jbnx4\"" Oct 30 21:40:34.934056 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.934033 6720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 21:40:34.934056 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.934055 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4mh2p\"" Oct 30 21:40:34.935082 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.935065 6720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 21:40:34.935190 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.935173 6720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 21:40:34.935438 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.935402 6720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 21:40:34.935689 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.935657 6720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 21:40:34.937522 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.937502 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pp8j7\"" Oct 30 21:40:34.937907 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.937894 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-ggbv4\"" Oct 30 21:40:34.937977 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.937964 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-pnbm4\"" Oct 30 21:40:34.938044 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.938027 6720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 21:40:34.938096 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.938057 6720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 21:40:34.938186 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.938172 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 21:40:34.938722 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.938707 6720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 21:40:34.938900 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.938883 6720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 21:40:34.939257 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.939238 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-n2fsk\"" Oct 30 21:40:34.939348 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.939262 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-csxns\"" Oct 30 21:40:34.939348 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.939289 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tzrj6\"" Oct 30 21:40:34.940112 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.940095 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 21:40:34.940170 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.940103 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-45g4h\"" Oct 30 21:40:34.940170 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.940119 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9c426\"" Oct 30 21:40:34.940243 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.940174 6720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 21:40:34.940280 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.940253 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 21:40:34.941168 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.941154 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Oct 30 21:40:34.941230 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.941170 6720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 21:40:34.941230 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.941183 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-glzqf\"" Oct 30 21:40:34.941230 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.941221 6720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 21:40:34.941349 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.941258 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-9shtx\"" Oct 30 21:40:34.941349 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:34.941318 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 21:40:35.017241 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.017217 6720 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-181.us-east-2.compute.internal" Oct 30 21:40:35.017775 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.017760 6720 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-181.us-east-2.compute.internal" Oct 30 21:40:35.023296 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.023278 6720 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 21:40:35.027599 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.027578 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 21:40:35.027819 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.027799 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 30 21:40:35.027913 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.027899 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 30 21:40:35.028261 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.028230 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 21:40:35.028324 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.028230 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 21:40:35.028324 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.028299 6720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 30 21:40:35.028446 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.028363 6720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 21:40:35.029480 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.029456 6720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 21:40:35.029480 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.029479 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 21:40:35.030155 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.030131 6720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 21:40:35.030243 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.030171 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Oct 30 21:40:35.030243 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.030193 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 21:40:35.031911 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.031893 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 30 21:40:35.032020 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.031999 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 21:40:35.032085 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.031904 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 30 21:40:35.032085 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.031930 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 30 21:40:35.032085 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.031937 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 30 21:40:35.032085 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.032043 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 21:40:35.032085 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.031986 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 30 21:40:35.032085 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.031904 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 30 21:40:35.032085 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.032079 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 21:40:35.032350 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.031979 6720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 21:40:35.032350 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.032055 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 21:40:35.032350 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.032026 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 21:40:35.032350 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.032165 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-6slhptju3coep\"" Oct 30 21:40:35.032350 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.032176 6720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 21:40:35.032567 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.032550 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 21:40:35.033888 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.033872 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-3d1aq256tb889\"" Oct 30 21:40:35.033969 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.033951 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 30 21:40:35.033969 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.033965 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-gd9hw\"" Oct 30 21:40:35.034072 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.033978 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-g4b65\"" Oct 30 21:40:35.034072 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.033985 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-9f5cm\"" Oct 30 21:40:35.034072 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.033995 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 30 21:40:35.034072 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.034003 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-f6sd2\"" Oct 30 21:40:35.034072 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.033995 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-rdbs4\"" Oct 30 21:40:35.034072 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.034015 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 30 21:40:35.034072 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.034024 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-9cpg6\"" Oct 30 21:40:35.034283 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.034236 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-ls765\"" Oct 30 21:40:35.034283 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.034249 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-host-slash\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:40:35.034283 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.034271 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-log-socket\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:40:35.034404 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.034294 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/90136025-abf7-4dc6-ac4b-c3050fc417f6-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6fd8fdc4c5-xlslr\" (UID: \"90136025-abf7-4dc6-ac4b-c3050fc417f6\") " pod="openshift-monitoring/thanos-querier-6fd8fdc4c5-xlslr" Oct 30 21:40:35.034404 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.034311 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/90136025-abf7-4dc6-ac4b-c3050fc417f6-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6fd8fdc4c5-xlslr\" (UID: \"90136025-abf7-4dc6-ac4b-c3050fc417f6\") " pod="openshift-monitoring/thanos-querier-6fd8fdc4c5-xlslr" Oct 30 21:40:35.034404 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.034327 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/970cd7bd-9018-4f3c-976a-af47bab1460a-etc-sysctl-conf\") pod \"tuned-vsgzd\" (UID: \"970cd7bd-9018-4f3c-976a-af47bab1460a\") " pod="openshift-cluster-node-tuning-operator/tuned-vsgzd" Oct 30 21:40:35.034404 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.034370 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlvnp\" (UniqueName: \"kubernetes.io/projected/e2c4958f-c7a5-4b6a-9b29-c7e2cd7d2d41-kube-api-access-zlvnp\") pod \"dns-default-v9jlm\" (UID: \"e2c4958f-c7a5-4b6a-9b29-c7e2cd7d2d41\") " pod="openshift-dns/dns-default-v9jlm" Oct 30 21:40:35.034404 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.034370 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 30 21:40:35.034577 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.034543 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62-sys\") pod \"node-exporter-hbg8k\" (UID: \"c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62\") " pod="openshift-monitoring/node-exporter-hbg8k" Oct 30 21:40:35.034628 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.034608 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/90136025-abf7-4dc6-ac4b-c3050fc417f6-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6fd8fdc4c5-xlslr\" (UID: \"90136025-abf7-4dc6-ac4b-c3050fc417f6\") " pod="openshift-monitoring/thanos-querier-6fd8fdc4c5-xlslr" Oct 30 21:40:35.034628 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.034618 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/90136025-abf7-4dc6-ac4b-c3050fc417f6-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6fd8fdc4c5-xlslr\" (UID: \"90136025-abf7-4dc6-ac4b-c3050fc417f6\") " pod="openshift-monitoring/thanos-querier-6fd8fdc4c5-xlslr" Oct 30 21:40:35.034727 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.034612 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7x77\" (UniqueName: \"kubernetes.io/projected/3018f90d-07fd-4d10-80a3-1f14248e638c-kube-api-access-k7x77\") pod \"ingress-canary-tj676\" (UID: \"3018f90d-07fd-4d10-80a3-1f14248e638c\") " pod="openshift-ingress-canary/ingress-canary-tj676" Oct 30 21:40:35.034727 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.034685 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-config\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:35.034727 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.034706 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/77670d00-a9cb-4607-bfb3-61ae91253cca-hosts-file\") pod \"node-resolver-pnfw6\" (UID: \"77670d00-a9cb-4607-bfb3-61ae91253cca\") " pod="openshift-dns/node-resolver-pnfw6" Oct 30 21:40:35.034727 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.034722 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7d451a16-523d-4b43-95cc-de9eeffd8402-system-cni-dir\") pod \"multus-additional-cni-plugins-j8vz9\" (UID: \"7d451a16-523d-4b43-95cc-de9eeffd8402\") " pod="openshift-multus/multus-additional-cni-plugins-j8vz9" Oct 30 21:40:35.034927 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.034751 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f4b104cb-8d0c-4519-86e0-07ea0e71a4a0-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-4xbzf\" (UID: \"f4b104cb-8d0c-4519-86e0-07ea0e71a4a0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4xbzf" Oct 30 21:40:35.034927 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.034781 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/90136025-abf7-4dc6-ac4b-c3050fc417f6-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6fd8fdc4c5-xlslr\" (UID: \"90136025-abf7-4dc6-ac4b-c3050fc417f6\") " pod="openshift-monitoring/thanos-querier-6fd8fdc4c5-xlslr" Oct 30 21:40:35.034927 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.034854 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ec3fe241-c753-4845-8b7a-0469c73e4c3b-service-ca\") pod \"console-6db5565879-ndj8n\" (UID: \"ec3fe241-c753-4845-8b7a-0469c73e4c3b\") " pod="openshift-console/console-6db5565879-ndj8n" Oct 30 21:40:35.034927 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.034893 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f4b104cb-8d0c-4519-86e0-07ea0e71a4a0-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-4xbzf\" (UID: \"f4b104cb-8d0c-4519-86e0-07ea0e71a4a0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4xbzf" Oct 30 21:40:35.035061 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:35.034927 6720 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Oct 30 21:40:35.035061 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.034936 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-config\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:35.035061 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.034947 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/970cd7bd-9018-4f3c-976a-af47bab1460a-lib-modules\") pod \"tuned-vsgzd\" (UID: \"970cd7bd-9018-4f3c-976a-af47bab1460a\") " pod="openshift-cluster-node-tuning-operator/tuned-vsgzd" Oct 30 21:40:35.035061 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.034982 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/90136025-abf7-4dc6-ac4b-c3050fc417f6-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6fd8fdc4c5-xlslr\" (UID: \"90136025-abf7-4dc6-ac4b-c3050fc417f6\") " pod="openshift-monitoring/thanos-querier-6fd8fdc4c5-xlslr" Oct 30 21:40:35.035061 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.034991 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/64995343-c3c1-4915-953a-d1e4a5252461-device-dir\") pod \"aws-ebs-csi-driver-node-kf9nx\" (UID: \"64995343-c3c1-4915-953a-d1e4a5252461\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kf9nx" Oct 30 21:40:35.035061 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:35.035013 6720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ec3fe241-c753-4845-8b7a-0469c73e4c3b-service-ca podName:ec3fe241-c753-4845-8b7a-0469c73e4c3b nodeName:}" failed. No retries permitted until 2026-10-30 21:40:35.534994451 +0000 UTC m=+1.705959645 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/ec3fe241-c753-4845-8b7a-0469c73e4c3b-service-ca") pod "console-6db5565879-ndj8n" (UID: "ec3fe241-c753-4845-8b7a-0469c73e4c3b") : object "openshift-console"/"service-ca" not registered Oct 30 21:40:35.035303 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.035083 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62-node-exporter-wtmp\") pod \"node-exporter-hbg8k\" (UID: \"c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62\") " pod="openshift-monitoring/node-exporter-hbg8k" Oct 30 21:40:35.035303 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.035102 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/971a083e-69b8-46d1-b56a-9ccf0f4d557d-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-44hwx\" (UID: \"971a083e-69b8-46d1-b56a-9ccf0f4d557d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-44hwx" Oct 30 21:40:35.035303 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.035117 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:35.035303 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.035132 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-host-run-netns\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:40:35.035303 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.035146 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-var-lib-openvswitch\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:40:35.035303 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.035165 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62-root\") pod \"node-exporter-hbg8k\" (UID: \"c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62\") " pod="openshift-monitoring/node-exporter-hbg8k" Oct 30 21:40:35.035303 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.035210 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/970cd7bd-9018-4f3c-976a-af47bab1460a-etc-systemd\") pod \"tuned-vsgzd\" (UID: \"970cd7bd-9018-4f3c-976a-af47bab1460a\") " pod="openshift-cluster-node-tuning-operator/tuned-vsgzd" Oct 30 21:40:35.035303 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.035248 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tpzc\" (UniqueName: \"kubernetes.io/projected/df51787b-87eb-420c-aa69-3d2762bdd386-kube-api-access-5tpzc\") pod \"network-check-target-9h6ck\" (UID: \"df51787b-87eb-420c-aa69-3d2762bdd386\") " pod="openshift-network-diagnostics/network-check-target-9h6ck" Oct 30 21:40:35.035303 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.035275 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:35.035743 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.035301 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ec3fe241-c753-4845-8b7a-0469c73e4c3b-console-config\") pod \"console-6db5565879-ndj8n\" (UID: \"ec3fe241-c753-4845-8b7a-0469c73e4c3b\") " pod="openshift-console/console-6db5565879-ndj8n" Oct 30 21:40:35.035743 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:35.035373 6720 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Oct 30 21:40:35.035743 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.035375 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:35.035743 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:35.035455 6720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ec3fe241-c753-4845-8b7a-0469c73e4c3b-console-config podName:ec3fe241-c753-4845-8b7a-0469c73e4c3b nodeName:}" failed. No retries permitted until 2026-10-30 21:40:35.535438238 +0000 UTC m=+1.706403401 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/ec3fe241-c753-4845-8b7a-0469c73e4c3b-console-config") pod "console-6db5565879-ndj8n" (UID: "ec3fe241-c753-4845-8b7a-0469c73e4c3b") : object "openshift-console"/"console-config" not registered Oct 30 21:40:35.035743 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.035494 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rczmc\" (UniqueName: \"kubernetes.io/projected/ec3fe241-c753-4845-8b7a-0469c73e4c3b-kube-api-access-rczmc\") pod \"console-6db5565879-ndj8n\" (UID: \"ec3fe241-c753-4845-8b7a-0469c73e4c3b\") " pod="openshift-console/console-6db5565879-ndj8n" Oct 30 21:40:35.035743 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.035514 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:35.035743 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.035530 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/64995343-c3c1-4915-953a-d1e4a5252461-etc-selinux\") pod \"aws-ebs-csi-driver-node-kf9nx\" (UID: \"64995343-c3c1-4915-953a-d1e4a5252461\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kf9nx" Oct 30 21:40:35.035743 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.035530 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:35.035743 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.035553 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/27f7c44f-0287-4608-a04a-7f108a25b121-agent-certs\") pod \"konnectivity-agent-jg4wr\" (UID: \"27f7c44f-0287-4608-a04a-7f108a25b121\") " pod="kube-system/konnectivity-agent-jg4wr" Oct 30 21:40:35.035743 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.035565 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-fhou7149cukh2\"" Oct 30 21:40:35.035743 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.035578 6720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 30 21:40:35.035743 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.035582 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7d451a16-523d-4b43-95cc-de9eeffd8402-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-j8vz9\" (UID: \"7d451a16-523d-4b43-95cc-de9eeffd8402\") " pod="openshift-multus/multus-additional-cni-plugins-j8vz9" Oct 30 21:40:35.035743 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.035607 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qchq\" (UniqueName: \"kubernetes.io/projected/90136025-abf7-4dc6-ac4b-c3050fc417f6-kube-api-access-6qchq\") pod \"thanos-querier-6fd8fdc4c5-xlslr\" (UID: \"90136025-abf7-4dc6-ac4b-c3050fc417f6\") " pod="openshift-monitoring/thanos-querier-6fd8fdc4c5-xlslr" Oct 30 21:40:35.035743 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.035612 6720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 21:40:35.035743 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.035652 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/568ba29e-ccc3-4a4c-9a4f-192c0b931b0d-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-hrkzk\" (UID: \"568ba29e-ccc3-4a4c-9a4f-192c0b931b0d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-hrkzk" Oct 30 21:40:35.035743 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.035690 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b8ed5eda-cbec-4adf-93b1-0ee0f1203f30-data-volume\") pod \"insights-runtime-extractor-bhdkg\" (UID: \"b8ed5eda-cbec-4adf-93b1-0ee0f1203f30\") " pod="openshift-insights/insights-runtime-extractor-bhdkg" Oct 30 21:40:35.035743 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.035697 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:35.035743 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.035732 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a2e2c1af-8ab2-49c6-91ac-4712b88e0571-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-qv74l\" (UID: \"a2e2c1af-8ab2-49c6-91ac-4712b88e0571\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-qv74l" Oct 30 21:40:35.036448 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.035761 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/17127db1-cf66-4951-ba5d-eb486ee9d3d5-kubelet-config\") pod \"global-pull-secret-syncer-vckqz\" (UID: \"17127db1-cf66-4951-ba5d-eb486ee9d3d5\") " pod="kube-system/global-pull-secret-syncer-vckqz" Oct 30 21:40:35.036448 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.035807 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7d451a16-523d-4b43-95cc-de9eeffd8402-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-j8vz9\" (UID: \"7d451a16-523d-4b43-95cc-de9eeffd8402\") " pod="openshift-multus/multus-additional-cni-plugins-j8vz9" Oct 30 21:40:35.036448 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.035851 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/17127db1-cf66-4951-ba5d-eb486ee9d3d5-dbus\") pod \"global-pull-secret-syncer-vckqz\" (UID: \"17127db1-cf66-4951-ba5d-eb486ee9d3d5\") " pod="kube-system/global-pull-secret-syncer-vckqz" Oct 30 21:40:35.036448 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.035973 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4-metrics-certs\") pod \"network-metrics-daemon-68gsj\" (UID: \"0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4\") " pod="openshift-multus/network-metrics-daemon-68gsj" Oct 30 21:40:35.036448 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.035994 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/568ba29e-ccc3-4a4c-9a4f-192c0b931b0d-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-hrkzk\" (UID: \"568ba29e-ccc3-4a4c-9a4f-192c0b931b0d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-hrkzk" Oct 30 21:40:35.036448 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.035991 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/27f7c44f-0287-4608-a04a-7f108a25b121-agent-certs\") pod \"konnectivity-agent-jg4wr\" (UID: \"27f7c44f-0287-4608-a04a-7f108a25b121\") " pod="kube-system/konnectivity-agent-jg4wr" Oct 30 21:40:35.036448 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.036003 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/64995343-c3c1-4915-953a-d1e4a5252461-socket-dir\") pod \"aws-ebs-csi-driver-node-kf9nx\" (UID: \"64995343-c3c1-4915-953a-d1e4a5252461\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kf9nx" Oct 30 21:40:35.036448 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.036001 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b8ed5eda-cbec-4adf-93b1-0ee0f1203f30-data-volume\") pod \"insights-runtime-extractor-bhdkg\" (UID: \"b8ed5eda-cbec-4adf-93b1-0ee0f1203f30\") " pod="openshift-insights/insights-runtime-extractor-bhdkg" Oct 30 21:40:35.036448 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.036031 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a2e2c1af-8ab2-49c6-91ac-4712b88e0571-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-qv74l\" (UID: \"a2e2c1af-8ab2-49c6-91ac-4712b88e0571\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-qv74l" Oct 30 21:40:35.036448 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.036043 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-host-cni-netd\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:40:35.036448 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.036076 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b8ed5eda-cbec-4adf-93b1-0ee0f1203f30-crio-socket\") pod \"insights-runtime-extractor-bhdkg\" (UID: \"b8ed5eda-cbec-4adf-93b1-0ee0f1203f30\") " pod="openshift-insights/insights-runtime-extractor-bhdkg" Oct 30 21:40:35.036448 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.036113 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/970cd7bd-9018-4f3c-976a-af47bab1460a-etc-modprobe-d\") pod \"tuned-vsgzd\" (UID: \"970cd7bd-9018-4f3c-976a-af47bab1460a\") " pod="openshift-cluster-node-tuning-operator/tuned-vsgzd" Oct 30 21:40:35.036448 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.036151 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/970cd7bd-9018-4f3c-976a-af47bab1460a-sys\") pod \"tuned-vsgzd\" (UID: \"970cd7bd-9018-4f3c-976a-af47bab1460a\") " pod="openshift-cluster-node-tuning-operator/tuned-vsgzd" Oct 30 21:40:35.036448 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.036187 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4-metrics-certs\") pod \"network-metrics-daemon-68gsj\" (UID: \"0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4\") " pod="openshift-multus/network-metrics-daemon-68gsj" Oct 30 21:40:35.036448 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.036198 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/970cd7bd-9018-4f3c-976a-af47bab1460a-host\") pod \"tuned-vsgzd\" (UID: \"970cd7bd-9018-4f3c-976a-af47bab1460a\") " pod="openshift-cluster-node-tuning-operator/tuned-vsgzd" Oct 30 21:40:35.036448 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.036222 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-host-var-lib-cni-bin\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:40:35.036448 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.036245 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2kzt\" (UniqueName: \"kubernetes.io/projected/970cd7bd-9018-4f3c-976a-af47bab1460a-kube-api-access-b2kzt\") pod \"tuned-vsgzd\" (UID: \"970cd7bd-9018-4f3c-976a-af47bab1460a\") " pod="openshift-cluster-node-tuning-operator/tuned-vsgzd" Oct 30 21:40:35.036448 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.036270 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d-secret-metrics-server-tls\") pod \"metrics-server-c74454f78-xph4m\" (UID: \"df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d\") " pod="openshift-monitoring/metrics-server-c74454f78-xph4m" Oct 30 21:40:35.036448 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.036326 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-config-out\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:35.036448 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.036431 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8b0ede8c-1c7c-479c-8c04-29d849fdbf43-host\") pod \"node-ca-m7qjt\" (UID: \"8b0ede8c-1c7c-479c-8c04-29d849fdbf43\") " pod="openshift-image-registry/node-ca-m7qjt" Oct 30 21:40:35.036448 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.036447 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-config-out\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:35.037251 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.036462 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c85rm\" (UniqueName: \"kubernetes.io/projected/64995343-c3c1-4915-953a-d1e4a5252461-kube-api-access-c85rm\") pod \"aws-ebs-csi-driver-node-kf9nx\" (UID: \"64995343-c3c1-4915-953a-d1e4a5252461\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kf9nx" Oct 30 21:40:35.037251 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.036496 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/568ba29e-ccc3-4a4c-9a4f-192c0b931b0d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-hrkzk\" (UID: \"568ba29e-ccc3-4a4c-9a4f-192c0b931b0d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-hrkzk" Oct 30 21:40:35.037251 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.036519 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d-secret-metrics-server-tls\") pod \"metrics-server-c74454f78-xph4m\" (UID: \"df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d\") " pod="openshift-monitoring/metrics-server-c74454f78-xph4m" Oct 30 21:40:35.037251 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.036522 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-host-kubelet\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:40:35.037251 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.036560 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62-metrics-client-ca\") pod \"node-exporter-hbg8k\" (UID: \"c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62\") " pod="openshift-monitoring/node-exporter-hbg8k" Oct 30 21:40:35.037251 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.036576 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/64995343-c3c1-4915-953a-d1e4a5252461-sys-fs\") pod \"aws-ebs-csi-driver-node-kf9nx\" (UID: \"64995343-c3c1-4915-953a-d1e4a5252461\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kf9nx" Oct 30 21:40:35.037251 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.036592 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d0a1535a-33cb-4a99-899d-aa8b3333326f-iptables-alerter-script\") pod \"iptables-alerter-bzhn4\" (UID: \"d0a1535a-33cb-4a99-899d-aa8b3333326f\") " pod="openshift-network-operator/iptables-alerter-bzhn4" Oct 30 21:40:35.037251 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.036609 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2c4958f-c7a5-4b6a-9b29-c7e2cd7d2d41-tmp-dir\") pod \"dns-default-v9jlm\" (UID: \"e2c4958f-c7a5-4b6a-9b29-c7e2cd7d2d41\") " pod="openshift-dns/dns-default-v9jlm" Oct 30 21:40:35.037251 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.036630 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/90136025-abf7-4dc6-ac4b-c3050fc417f6-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6fd8fdc4c5-xlslr\" (UID: \"90136025-abf7-4dc6-ac4b-c3050fc417f6\") " pod="openshift-monitoring/thanos-querier-6fd8fdc4c5-xlslr" Oct 30 21:40:35.037251 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.036653 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-hostroot\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:40:35.037251 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.036695 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l725d\" (UniqueName: \"kubernetes.io/projected/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-kube-api-access-l725d\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:40:35.037251 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.036731 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/27f7c44f-0287-4608-a04a-7f108a25b121-konnectivity-ca\") pod \"konnectivity-agent-jg4wr\" (UID: \"27f7c44f-0287-4608-a04a-7f108a25b121\") " pod="kube-system/konnectivity-agent-jg4wr" Oct 30 21:40:35.037251 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.036734 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2c4958f-c7a5-4b6a-9b29-c7e2cd7d2d41-tmp-dir\") pod \"dns-default-v9jlm\" (UID: \"e2c4958f-c7a5-4b6a-9b29-c7e2cd7d2d41\") " pod="openshift-dns/dns-default-v9jlm" Oct 30 21:40:35.037251 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.036760 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-ovnkube-script-lib\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:40:35.037251 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.036766 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/568ba29e-ccc3-4a4c-9a4f-192c0b931b0d-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-hrkzk\" (UID: \"568ba29e-ccc3-4a4c-9a4f-192c0b931b0d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-hrkzk" Oct 30 21:40:35.037251 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.036773 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62-metrics-client-ca\") pod \"node-exporter-hbg8k\" (UID: \"c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62\") " pod="openshift-monitoring/node-exporter-hbg8k" Oct 30 21:40:35.037251 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.036792 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-host-run-multus-certs\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:40:35.037251 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.036832 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ldlh\" (UniqueName: \"kubernetes.io/projected/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-kube-api-access-4ldlh\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:35.037251 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.036913 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/90136025-abf7-4dc6-ac4b-c3050fc417f6-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6fd8fdc4c5-xlslr\" (UID: \"90136025-abf7-4dc6-ac4b-c3050fc417f6\") " pod="openshift-monitoring/thanos-querier-6fd8fdc4c5-xlslr" Oct 30 21:40:35.037251 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.036914 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d0a1535a-33cb-4a99-899d-aa8b3333326f-iptables-alerter-script\") pod \"iptables-alerter-bzhn4\" (UID: \"d0a1535a-33cb-4a99-899d-aa8b3333326f\") " pod="openshift-network-operator/iptables-alerter-bzhn4" Oct 30 21:40:35.037251 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.036935 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/27f7c44f-0287-4608-a04a-7f108a25b121-konnectivity-ca\") pod \"konnectivity-agent-jg4wr\" (UID: \"27f7c44f-0287-4608-a04a-7f108a25b121\") " pod="kube-system/konnectivity-agent-jg4wr" Oct 30 21:40:35.037251 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.036948 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-etc-openvswitch\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:40:35.037251 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.036985 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-host-run-ovn-kubernetes\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:40:35.037251 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.037011 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-etc-kubernetes\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:40:35.037251 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.037041 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wv4r\" (UniqueName: \"kubernetes.io/projected/568ba29e-ccc3-4a4c-9a4f-192c0b931b0d-kube-api-access-5wv4r\") pod \"prometheus-operator-9cd89b74c-hrkzk\" (UID: \"568ba29e-ccc3-4a4c-9a4f-192c0b931b0d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-hrkzk" Oct 30 21:40:35.037251 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.037059 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-ovnkube-script-lib\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:40:35.037251 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.037079 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f4b104cb-8d0c-4519-86e0-07ea0e71a4a0-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-4xbzf\" (UID: \"f4b104cb-8d0c-4519-86e0-07ea0e71a4a0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4xbzf" Oct 30 21:40:35.037251 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.037133 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:35.038533 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.037277 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f4b104cb-8d0c-4519-86e0-07ea0e71a4a0-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-4xbzf\" (UID: \"f4b104cb-8d0c-4519-86e0-07ea0e71a4a0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4xbzf" Oct 30 21:40:35.038533 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.037292 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-env-overrides\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:40:35.038533 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.037335 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm68d\" (UniqueName: \"kubernetes.io/projected/b8ed5eda-cbec-4adf-93b1-0ee0f1203f30-kube-api-access-gm68d\") pod \"insights-runtime-extractor-bhdkg\" (UID: \"b8ed5eda-cbec-4adf-93b1-0ee0f1203f30\") " pod="openshift-insights/insights-runtime-extractor-bhdkg" Oct 30 21:40:35.038533 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.037370 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62-node-exporter-accelerators-collector-config\") pod \"node-exporter-hbg8k\" (UID: \"c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62\") " pod="openshift-monitoring/node-exporter-hbg8k" Oct 30 21:40:35.038533 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.037426 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-system-cni-dir\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:40:35.038533 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.037455 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-cnibin\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:40:35.038533 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.037461 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:35.038533 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.037484 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ec3fe241-c753-4845-8b7a-0469c73e4c3b-oauth-serving-cert\") pod \"console-6db5565879-ndj8n\" (UID: \"ec3fe241-c753-4845-8b7a-0469c73e4c3b\") " pod="openshift-console/console-6db5565879-ndj8n" Oct 30 21:40:35.038533 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.037498 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-env-overrides\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:40:35.038533 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.037525 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:35.038533 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:35.037578 6720 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Oct 30 21:40:35.038533 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.037579 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77670d00-a9cb-4607-bfb3-61ae91253cca-tmp-dir\") pod \"node-resolver-pnfw6\" (UID: \"77670d00-a9cb-4607-bfb3-61ae91253cca\") " pod="openshift-dns/node-resolver-pnfw6" Oct 30 21:40:35.038533 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.037637 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:35.038533 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.037651 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62-node-exporter-accelerators-collector-config\") pod \"node-exporter-hbg8k\" (UID: \"c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62\") " pod="openshift-monitoring/node-exporter-hbg8k" Oct 30 21:40:35.038533 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.037718 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/568ba29e-ccc3-4a4c-9a4f-192c0b931b0d-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-hrkzk\" (UID: \"568ba29e-ccc3-4a4c-9a4f-192c0b931b0d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-hrkzk" Oct 30 21:40:35.038533 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.037732 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77670d00-a9cb-4607-bfb3-61ae91253cca-tmp-dir\") pod \"node-resolver-pnfw6\" (UID: \"77670d00-a9cb-4607-bfb3-61ae91253cca\") " pod="openshift-dns/node-resolver-pnfw6" Oct 30 21:40:35.038533 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:35.037754 6720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ec3fe241-c753-4845-8b7a-0469c73e4c3b-oauth-serving-cert podName:ec3fe241-c753-4845-8b7a-0469c73e4c3b nodeName:}" failed. No retries permitted until 2026-10-30 21:40:35.537741602 +0000 UTC m=+1.708706786 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/ec3fe241-c753-4845-8b7a-0469c73e4c3b-oauth-serving-cert") pod "console-6db5565879-ndj8n" (UID: "ec3fe241-c753-4845-8b7a-0469c73e4c3b") : object "openshift-console"/"oauth-serving-cert" not registered Oct 30 21:40:35.038533 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.037771 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:35.038533 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.037791 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-284h6\" (UniqueName: \"kubernetes.io/projected/7d451a16-523d-4b43-95cc-de9eeffd8402-kube-api-access-284h6\") pod \"multus-additional-cni-plugins-j8vz9\" (UID: \"7d451a16-523d-4b43-95cc-de9eeffd8402\") " pod="openshift-multus/multus-additional-cni-plugins-j8vz9" Oct 30 21:40:35.038533 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.037819 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-systemd-units\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:40:35.038533 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.037835 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-node-log\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:40:35.038533 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.037892 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d-client-ca-bundle\") pod \"metrics-server-c74454f78-xph4m\" (UID: \"df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d\") " pod="openshift-monitoring/metrics-server-c74454f78-xph4m" Oct 30 21:40:35.038533 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.037919 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/568ba29e-ccc3-4a4c-9a4f-192c0b931b0d-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-hrkzk\" (UID: \"568ba29e-ccc3-4a4c-9a4f-192c0b931b0d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-hrkzk" Oct 30 21:40:35.038533 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.037920 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvrrj\" (UniqueName: \"kubernetes.io/projected/df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d-kube-api-access-bvrrj\") pod \"metrics-server-c74454f78-xph4m\" (UID: \"df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d\") " pod="openshift-monitoring/metrics-server-c74454f78-xph4m" Oct 30 21:40:35.038533 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.037969 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d0a1535a-33cb-4a99-899d-aa8b3333326f-host-slash\") pod \"iptables-alerter-bzhn4\" (UID: \"d0a1535a-33cb-4a99-899d-aa8b3333326f\") " pod="openshift-network-operator/iptables-alerter-bzhn4" Oct 30 21:40:35.038533 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.037998 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj7w5\" (UniqueName: \"kubernetes.io/projected/d0a1535a-33cb-4a99-899d-aa8b3333326f-kube-api-access-sj7w5\") pod \"iptables-alerter-bzhn4\" (UID: \"d0a1535a-33cb-4a99-899d-aa8b3333326f\") " pod="openshift-network-operator/iptables-alerter-bzhn4" Oct 30 21:40:35.038533 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.038026 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-run-openvswitch\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:40:35.038533 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.038052 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62-node-exporter-tls\") pod \"node-exporter-hbg8k\" (UID: \"c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62\") " pod="openshift-monitoring/node-exporter-hbg8k" Oct 30 21:40:35.038533 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.038076 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-cni-binary-copy\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:40:35.038533 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.038084 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:35.038533 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.038104 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d-secret-metrics-server-client-certs\") pod \"metrics-server-c74454f78-xph4m\" (UID: \"df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d\") " pod="openshift-monitoring/metrics-server-c74454f78-xph4m" Oct 30 21:40:35.038533 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.038145 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:35.038533 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.038149 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d-client-ca-bundle\") pod \"metrics-server-c74454f78-xph4m\" (UID: \"df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d\") " pod="openshift-monitoring/metrics-server-c74454f78-xph4m" Oct 30 21:40:35.038533 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.038282 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/90136025-abf7-4dc6-ac4b-c3050fc417f6-secret-thanos-querier-tls\") pod \"thanos-querier-6fd8fdc4c5-xlslr\" (UID: \"90136025-abf7-4dc6-ac4b-c3050fc417f6\") " pod="openshift-monitoring/thanos-querier-6fd8fdc4c5-xlslr" Oct 30 21:40:35.038533 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.038306 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d-secret-metrics-server-client-certs\") pod \"metrics-server-c74454f78-xph4m\" (UID: \"df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d\") " pod="openshift-monitoring/metrics-server-c74454f78-xph4m" Oct 30 21:40:35.038533 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.038332 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/90136025-abf7-4dc6-ac4b-c3050fc417f6-metrics-client-ca\") pod \"thanos-querier-6fd8fdc4c5-xlslr\" (UID: \"90136025-abf7-4dc6-ac4b-c3050fc417f6\") " pod="openshift-monitoring/thanos-querier-6fd8fdc4c5-xlslr" Oct 30 21:40:35.038533 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.038367 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-cni-binary-copy\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:40:35.038533 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.038361 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62-node-exporter-tls\") pod \"node-exporter-hbg8k\" (UID: \"c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62\") " pod="openshift-monitoring/node-exporter-hbg8k" Oct 30 21:40:35.038533 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.038435 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-run-systemd\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:40:35.038533 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.038519 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/970cd7bd-9018-4f3c-976a-af47bab1460a-var-lib-kubelet\") pod \"tuned-vsgzd\" (UID: \"970cd7bd-9018-4f3c-976a-af47bab1460a\") " pod="openshift-cluster-node-tuning-operator/tuned-vsgzd" Oct 30 21:40:35.038533 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.038549 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/970cd7bd-9018-4f3c-976a-af47bab1460a-etc-tuned\") pod \"tuned-vsgzd\" (UID: \"970cd7bd-9018-4f3c-976a-af47bab1460a\") " pod="openshift-cluster-node-tuning-operator/tuned-vsgzd" Oct 30 21:40:35.038533 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.038559 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/90136025-abf7-4dc6-ac4b-c3050fc417f6-secret-thanos-querier-tls\") pod \"thanos-querier-6fd8fdc4c5-xlslr\" (UID: \"90136025-abf7-4dc6-ac4b-c3050fc417f6\") " pod="openshift-monitoring/thanos-querier-6fd8fdc4c5-xlslr" Oct 30 21:40:35.038533 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.038563 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/90136025-abf7-4dc6-ac4b-c3050fc417f6-metrics-client-ca\") pod \"thanos-querier-6fd8fdc4c5-xlslr\" (UID: \"90136025-abf7-4dc6-ac4b-c3050fc417f6\") " pod="openshift-monitoring/thanos-querier-6fd8fdc4c5-xlslr" Oct 30 21:40:35.040378 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.038575 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ec3fe241-c753-4845-8b7a-0469c73e4c3b-console-serving-cert\") pod \"console-6db5565879-ndj8n\" (UID: \"ec3fe241-c753-4845-8b7a-0469c73e4c3b\") " pod="openshift-console/console-6db5565879-ndj8n" Oct 30 21:40:35.040378 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.038608 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-c74454f78-xph4m\" (UID: \"df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d\") " pod="openshift-monitoring/metrics-server-c74454f78-xph4m" Oct 30 21:40:35.040378 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:35.038675 6720 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Oct 30 21:40:35.040378 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.038675 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/64995343-c3c1-4915-953a-d1e4a5252461-registration-dir\") pod \"aws-ebs-csi-driver-node-kf9nx\" (UID: \"64995343-c3c1-4915-953a-d1e4a5252461\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kf9nx" Oct 30 21:40:35.040378 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.038701 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/970cd7bd-9018-4f3c-976a-af47bab1460a-etc-tuned\") pod \"tuned-vsgzd\" (UID: \"970cd7bd-9018-4f3c-976a-af47bab1460a\") " pod="openshift-cluster-node-tuning-operator/tuned-vsgzd" Oct 30 21:40:35.040378 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:35.038803 6720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec3fe241-c753-4845-8b7a-0469c73e4c3b-console-serving-cert podName:ec3fe241-c753-4845-8b7a-0469c73e4c3b nodeName:}" failed. No retries permitted until 2026-10-30 21:40:35.53878839 +0000 UTC m=+1.709753584 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/ec3fe241-c753-4845-8b7a-0469c73e4c3b-console-serving-cert") pod "console-6db5565879-ndj8n" (UID: "ec3fe241-c753-4845-8b7a-0469c73e4c3b") : object "openshift-console"/"console-serving-cert" not registered Oct 30 21:40:35.040378 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.038811 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-c74454f78-xph4m\" (UID: \"df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d\") " pod="openshift-monitoring/metrics-server-c74454f78-xph4m" Oct 30 21:40:35.040378 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.038841 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f4b104cb-8d0c-4519-86e0-07ea0e71a4a0-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-4xbzf\" (UID: \"f4b104cb-8d0c-4519-86e0-07ea0e71a4a0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4xbzf" Oct 30 21:40:35.040378 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.038879 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-host-var-lib-kubelet\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:40:35.040378 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.038915 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/970cd7bd-9018-4f3c-976a-af47bab1460a-run\") pod \"tuned-vsgzd\" (UID: \"970cd7bd-9018-4f3c-976a-af47bab1460a\") " pod="openshift-cluster-node-tuning-operator/tuned-vsgzd" Oct 30 21:40:35.040378 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.038946 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9874c\" (UniqueName: \"kubernetes.io/projected/a2e2c1af-8ab2-49c6-91ac-4712b88e0571-kube-api-access-9874c\") pod \"monitoring-plugin-77f4f76594-qv74l\" (UID: \"a2e2c1af-8ab2-49c6-91ac-4712b88e0571\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-qv74l" Oct 30 21:40:35.040378 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.038971 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d-audit-log\") pod \"metrics-server-c74454f78-xph4m\" (UID: \"df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d\") " pod="openshift-monitoring/metrics-server-c74454f78-xph4m" Oct 30 21:40:35.040378 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.039005 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f4b104cb-8d0c-4519-86e0-07ea0e71a4a0-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-4xbzf\" (UID: \"f4b104cb-8d0c-4519-86e0-07ea0e71a4a0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4xbzf" Oct 30 21:40:35.040378 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.039026 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:40:35.040378 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.039043 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e2c4958f-c7a5-4b6a-9b29-c7e2cd7d2d41-config-volume\") pod \"dns-default-v9jlm\" (UID: \"e2c4958f-c7a5-4b6a-9b29-c7e2cd7d2d41\") " pod="openshift-dns/dns-default-v9jlm" Oct 30 21:40:35.040378 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.039058 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hbg8k\" (UID: \"c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62\") " pod="openshift-monitoring/node-exporter-hbg8k" Oct 30 21:40:35.040378 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.039081 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/970cd7bd-9018-4f3c-976a-af47bab1460a-etc-sysctl-d\") pod \"tuned-vsgzd\" (UID: \"970cd7bd-9018-4f3c-976a-af47bab1460a\") " pod="openshift-cluster-node-tuning-operator/tuned-vsgzd" Oct 30 21:40:35.040378 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.039074 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f4b104cb-8d0c-4519-86e0-07ea0e71a4a0-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-4xbzf\" (UID: \"f4b104cb-8d0c-4519-86e0-07ea0e71a4a0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4xbzf" Oct 30 21:40:35.040378 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.039148 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:35.040378 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.039161 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d-audit-log\") pod \"metrics-server-c74454f78-xph4m\" (UID: \"df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d\") " pod="openshift-monitoring/metrics-server-c74454f78-xph4m" Oct 30 21:40:35.040378 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.039173 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b8ed5eda-cbec-4adf-93b1-0ee0f1203f30-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bhdkg\" (UID: \"b8ed5eda-cbec-4adf-93b1-0ee0f1203f30\") " pod="openshift-insights/insights-runtime-extractor-bhdkg" Oct 30 21:40:35.040378 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.039200 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-host-var-lib-cni-multus\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:40:35.040378 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.039245 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-multus-conf-dir\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:40:35.040378 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.039249 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e2c4958f-c7a5-4b6a-9b29-c7e2cd7d2d41-config-volume\") pod \"dns-default-v9jlm\" (UID: \"e2c4958f-c7a5-4b6a-9b29-c7e2cd7d2d41\") " pod="openshift-dns/dns-default-v9jlm" Oct 30 21:40:35.040378 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.039273 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hbg8k\" (UID: \"c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62\") " pod="openshift-monitoring/node-exporter-hbg8k" Oct 30 21:40:35.040378 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.039285 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/970cd7bd-9018-4f3c-976a-af47bab1460a-etc-sysconfig\") pod \"tuned-vsgzd\" (UID: \"970cd7bd-9018-4f3c-976a-af47bab1460a\") " pod="openshift-cluster-node-tuning-operator/tuned-vsgzd" Oct 30 21:40:35.040378 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.039276 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f4b104cb-8d0c-4519-86e0-07ea0e71a4a0-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-4xbzf\" (UID: \"f4b104cb-8d0c-4519-86e0-07ea0e71a4a0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4xbzf" Oct 30 21:40:35.040378 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.039316 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17127db1-cf66-4951-ba5d-eb486ee9d3d5-original-pull-secret\") pod \"global-pull-secret-syncer-vckqz\" (UID: \"17127db1-cf66-4951-ba5d-eb486ee9d3d5\") " pod="kube-system/global-pull-secret-syncer-vckqz" Oct 30 21:40:35.040378 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.039361 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d-metrics-server-audit-profiles\") pod \"metrics-server-c74454f78-xph4m\" (UID: \"df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d\") " pod="openshift-monitoring/metrics-server-c74454f78-xph4m" Oct 30 21:40:35.040378 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.039410 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:35.040378 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.039414 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b8ed5eda-cbec-4adf-93b1-0ee0f1203f30-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bhdkg\" (UID: \"b8ed5eda-cbec-4adf-93b1-0ee0f1203f30\") " pod="openshift-insights/insights-runtime-extractor-bhdkg" Oct 30 21:40:35.040378 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.039580 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:35.040378 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.039587 6720 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:40:35.040378 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.039610 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/90136025-abf7-4dc6-ac4b-c3050fc417f6-secret-grpc-tls\") pod \"thanos-querier-6fd8fdc4c5-xlslr\" (UID: \"90136025-abf7-4dc6-ac4b-c3050fc417f6\") " pod="openshift-monitoring/thanos-querier-6fd8fdc4c5-xlslr" Oct 30 21:40:35.040378 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.039635 6720 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:40:35.040378 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.039637 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:35.040378 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:35.039665 6720 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-137-181.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-181.us-east-2.compute.internal" Oct 30 21:40:35.040378 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:35.039680 6720 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-137-181.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-181.us-east-2.compute.internal" Oct 30 21:40:35.040378 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.039637 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-multus-cni-dir\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:40:35.040378 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.039746 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sl5w\" (UniqueName: \"kubernetes.io/projected/8b0ede8c-1c7c-479c-8c04-29d849fdbf43-kube-api-access-6sl5w\") pod \"node-ca-m7qjt\" (UID: \"8b0ede8c-1c7c-479c-8c04-29d849fdbf43\") " pod="openshift-image-registry/node-ca-m7qjt" Oct 30 21:40:35.040378 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.039772 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7d451a16-523d-4b43-95cc-de9eeffd8402-cni-binary-copy\") pod \"multus-additional-cni-plugins-j8vz9\" (UID: \"7d451a16-523d-4b43-95cc-de9eeffd8402\") " pod="openshift-multus/multus-additional-cni-plugins-j8vz9" Oct 30 21:40:35.040378 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.039789 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:35.040378 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.039761 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d-metrics-server-audit-profiles\") pod \"metrics-server-c74454f78-xph4m\" (UID: \"df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d\") " pod="openshift-monitoring/metrics-server-c74454f78-xph4m" Oct 30 21:40:35.040378 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.039804 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/17127db1-cf66-4951-ba5d-eb486ee9d3d5-original-pull-secret\") pod \"global-pull-secret-syncer-vckqz\" (UID: \"17127db1-cf66-4951-ba5d-eb486ee9d3d5\") " pod="kube-system/global-pull-secret-syncer-vckqz" Oct 30 21:40:35.040378 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.039801 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7d451a16-523d-4b43-95cc-de9eeffd8402-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-j8vz9\" (UID: \"7d451a16-523d-4b43-95cc-de9eeffd8402\") " pod="openshift-multus/multus-additional-cni-plugins-j8vz9" Oct 30 21:40:35.040378 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.039861 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-ovnkube-config\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:40:35.040378 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.039887 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2c4958f-c7a5-4b6a-9b29-c7e2cd7d2d41-metrics-tls\") pod \"dns-default-v9jlm\" (UID: \"e2c4958f-c7a5-4b6a-9b29-c7e2cd7d2d41\") " pod="openshift-dns/dns-default-v9jlm" Oct 30 21:40:35.040378 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.039914 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbcgc\" (UniqueName: \"kubernetes.io/projected/0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4-kube-api-access-pbcgc\") pod \"network-metrics-daemon-68gsj\" (UID: \"0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4\") " pod="openshift-multus/network-metrics-daemon-68gsj" Oct 30 21:40:35.040378 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.039941 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:35.040378 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.039976 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:35.040378 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.039986 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7d451a16-523d-4b43-95cc-de9eeffd8402-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-j8vz9\" (UID: \"7d451a16-523d-4b43-95cc-de9eeffd8402\") " pod="openshift-multus/multus-additional-cni-plugins-j8vz9" Oct 30 21:40:35.040378 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.040004 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/64995343-c3c1-4915-953a-d1e4a5252461-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kf9nx\" (UID: \"64995343-c3c1-4915-953a-d1e4a5252461\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kf9nx" Oct 30 21:40:35.040378 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.040025 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7d451a16-523d-4b43-95cc-de9eeffd8402-cni-binary-copy\") pod \"multus-additional-cni-plugins-j8vz9\" (UID: \"7d451a16-523d-4b43-95cc-de9eeffd8402\") " pod="openshift-multus/multus-additional-cni-plugins-j8vz9" Oct 30 21:40:35.040378 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.040034 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-run-ovn\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:40:35.040378 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.040035 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/90136025-abf7-4dc6-ac4b-c3050fc417f6-secret-grpc-tls\") pod \"thanos-querier-6fd8fdc4c5-xlslr\" (UID: \"90136025-abf7-4dc6-ac4b-c3050fc417f6\") " pod="openshift-monitoring/thanos-querier-6fd8fdc4c5-xlslr" Oct 30 21:40:35.040378 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.040088 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-web-config\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:35.040378 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.040167 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e2c4958f-c7a5-4b6a-9b29-c7e2cd7d2d41-metrics-tls\") pod \"dns-default-v9jlm\" (UID: \"e2c4958f-c7a5-4b6a-9b29-c7e2cd7d2d41\") " pod="openshift-dns/dns-default-v9jlm" Oct 30 21:40:35.040378 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.040173 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:35.040378 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.040214 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-host-cni-bin\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:40:35.040378 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.040214 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-ovnkube-config\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:40:35.040378 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.040264 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:35.040378 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.040265 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/971a083e-69b8-46d1-b56a-9ccf0f4d557d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-44hwx\" (UID: \"971a083e-69b8-46d1-b56a-9ccf0f4d557d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-44hwx" Oct 30 21:40:35.040378 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.040301 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ec3fe241-c753-4845-8b7a-0469c73e4c3b-console-oauth-config\") pod \"console-6db5565879-ndj8n\" (UID: \"ec3fe241-c753-4845-8b7a-0469c73e4c3b\") " pod="openshift-console/console-6db5565879-ndj8n" Oct 30 21:40:35.040378 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.040304 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-web-config\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:35.040378 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.040377 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:35.043053 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:35.040457 6720 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Oct 30 21:40:35.043053 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.040465 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f4b104cb-8d0c-4519-86e0-07ea0e71a4a0-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-4xbzf\" (UID: \"f4b104cb-8d0c-4519-86e0-07ea0e71a4a0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4xbzf" Oct 30 21:40:35.043053 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.040486 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/971a083e-69b8-46d1-b56a-9ccf0f4d557d-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-44hwx\" (UID: \"971a083e-69b8-46d1-b56a-9ccf0f4d557d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-44hwx" Oct 30 21:40:35.043053 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.040499 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-ovn-node-metrics-cert\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:40:35.043053 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.040526 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpmtf\" (UniqueName: \"kubernetes.io/projected/c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62-kube-api-access-qpmtf\") pod \"node-exporter-hbg8k\" (UID: \"c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62\") " pod="openshift-monitoring/node-exporter-hbg8k" Oct 30 21:40:35.043053 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.040553 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/970cd7bd-9018-4f3c-976a-af47bab1460a-etc-kubernetes\") pod \"tuned-vsgzd\" (UID: \"970cd7bd-9018-4f3c-976a-af47bab1460a\") " pod="openshift-cluster-node-tuning-operator/tuned-vsgzd" Oct 30 21:40:35.043053 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:35.040563 6720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec3fe241-c753-4845-8b7a-0469c73e4c3b-console-oauth-config podName:ec3fe241-c753-4845-8b7a-0469c73e4c3b nodeName:}" failed. No retries permitted until 2026-10-30 21:40:35.540548747 +0000 UTC m=+1.711513925 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/ec3fe241-c753-4845-8b7a-0469c73e4c3b-console-oauth-config") pod "console-6db5565879-ndj8n" (UID: "ec3fe241-c753-4845-8b7a-0469c73e4c3b") : object "openshift-console"/"console-oauth-config" not registered Oct 30 21:40:35.043053 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.040592 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/970cd7bd-9018-4f3c-976a-af47bab1460a-tmp\") pod \"tuned-vsgzd\" (UID: \"970cd7bd-9018-4f3c-976a-af47bab1460a\") " pod="openshift-cluster-node-tuning-operator/tuned-vsgzd" Oct 30 21:40:35.043053 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.040608 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:35.043053 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.040621 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfw8c\" (UniqueName: \"kubernetes.io/projected/971a083e-69b8-46d1-b56a-9ccf0f4d557d-kube-api-access-vfw8c\") pod \"openshift-state-metrics-7bd7489957-44hwx\" (UID: \"971a083e-69b8-46d1-b56a-9ccf0f4d557d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-44hwx" Oct 30 21:40:35.043053 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.040650 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:35.043053 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.040668 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/970cd7bd-9018-4f3c-976a-af47bab1460a-tmp\") pod \"tuned-vsgzd\" (UID: \"970cd7bd-9018-4f3c-976a-af47bab1460a\") " pod="openshift-cluster-node-tuning-operator/tuned-vsgzd" Oct 30 21:40:35.043053 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.040675 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3018f90d-07fd-4d10-80a3-1f14248e638c-cert\") pod \"ingress-canary-tj676\" (UID: \"3018f90d-07fd-4d10-80a3-1f14248e638c\") " pod="openshift-ingress-canary/ingress-canary-tj676" Oct 30 21:40:35.043053 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.040686 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f4b104cb-8d0c-4519-86e0-07ea0e71a4a0-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-4xbzf\" (UID: \"f4b104cb-8d0c-4519-86e0-07ea0e71a4a0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4xbzf" Oct 30 21:40:35.043053 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.040699 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8b0ede8c-1c7c-479c-8c04-29d849fdbf43-serviceca\") pod \"node-ca-m7qjt\" (UID: \"8b0ede8c-1c7c-479c-8c04-29d849fdbf43\") " pod="openshift-image-registry/node-ca-m7qjt" Oct 30 21:40:35.043053 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.040725 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-ovn-node-metrics-cert\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:40:35.043053 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.040724 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2stzq\" (UniqueName: \"kubernetes.io/projected/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-kube-api-access-2stzq\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:40:35.043053 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.040753 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/90136025-abf7-4dc6-ac4b-c3050fc417f6-tmp\") pod \"thanos-querier-6fd8fdc4c5-xlslr\" (UID: \"90136025-abf7-4dc6-ac4b-c3050fc417f6\") " pod="openshift-monitoring/thanos-querier-6fd8fdc4c5-xlslr" Oct 30 21:40:35.043053 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.040768 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-multus-socket-dir-parent\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:40:35.043053 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.040787 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-multus-daemon-config\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:40:35.043053 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.040816 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/971a083e-69b8-46d1-b56a-9ccf0f4d557d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-44hwx\" (UID: \"971a083e-69b8-46d1-b56a-9ccf0f4d557d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-44hwx" Oct 30 21:40:35.043053 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.040831 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ec3fe241-c753-4845-8b7a-0469c73e4c3b-tmp\") pod \"console-6db5565879-ndj8n\" (UID: \"ec3fe241-c753-4845-8b7a-0469c73e4c3b\") " pod="openshift-console/console-6db5565879-ndj8n" Oct 30 21:40:35.043053 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.040846 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:35.043053 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.040859 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:35.043053 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.040867 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:35.043053 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.040892 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62-node-exporter-textfile\") pod \"node-exporter-hbg8k\" (UID: \"c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62\") " pod="openshift-monitoring/node-exporter-hbg8k" Oct 30 21:40:35.043053 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.040919 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgq7t\" (UniqueName: \"kubernetes.io/projected/f4b104cb-8d0c-4519-86e0-07ea0e71a4a0-kube-api-access-wgq7t\") pod \"kube-state-metrics-5dc8d7d89b-4xbzf\" (UID: \"f4b104cb-8d0c-4519-86e0-07ea0e71a4a0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4xbzf" Oct 30 21:40:35.043053 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.040954 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b8ed5eda-cbec-4adf-93b1-0ee0f1203f30-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bhdkg\" (UID: \"b8ed5eda-cbec-4adf-93b1-0ee0f1203f30\") " pod="openshift-insights/insights-runtime-extractor-bhdkg" Oct 30 21:40:35.043053 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.040978 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3018f90d-07fd-4d10-80a3-1f14248e638c-cert\") pod \"ingress-canary-tj676\" (UID: \"3018f90d-07fd-4d10-80a3-1f14248e638c\") " pod="openshift-ingress-canary/ingress-canary-tj676" Oct 30 21:40:35.043053 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.040982 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-host-run-k8s-cni-cncf-io\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:40:35.043053 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.041020 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-host-run-netns\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:40:35.043053 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.041093 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/90136025-abf7-4dc6-ac4b-c3050fc417f6-tmp\") pod \"thanos-querier-6fd8fdc4c5-xlslr\" (UID: \"90136025-abf7-4dc6-ac4b-c3050fc417f6\") " pod="openshift-monitoring/thanos-querier-6fd8fdc4c5-xlslr" Oct 30 21:40:35.043053 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.041101 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62-node-exporter-textfile\") pod \"node-exporter-hbg8k\" (UID: \"c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62\") " pod="openshift-monitoring/node-exporter-hbg8k" Oct 30 21:40:35.043053 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.041130 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7d451a16-523d-4b43-95cc-de9eeffd8402-cnibin\") pod \"multus-additional-cni-plugins-j8vz9\" (UID: \"7d451a16-523d-4b43-95cc-de9eeffd8402\") " pod="openshift-multus/multus-additional-cni-plugins-j8vz9" Oct 30 21:40:35.043053 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.041149 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgrp4\" (UniqueName: \"kubernetes.io/projected/77670d00-a9cb-4607-bfb3-61ae91253cca-kube-api-access-mgrp4\") pod \"node-resolver-pnfw6\" (UID: \"77670d00-a9cb-4607-bfb3-61ae91253cca\") " pod="openshift-dns/node-resolver-pnfw6" Oct 30 21:40:35.043053 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.041165 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7d451a16-523d-4b43-95cc-de9eeffd8402-tuning-conf-dir\") pod \"multus-additional-cni-plugins-j8vz9\" (UID: \"7d451a16-523d-4b43-95cc-de9eeffd8402\") " pod="openshift-multus/multus-additional-cni-plugins-j8vz9" Oct 30 21:40:35.043053 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.041175 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ec3fe241-c753-4845-8b7a-0469c73e4c3b-tmp\") pod \"console-6db5565879-ndj8n\" (UID: \"ec3fe241-c753-4845-8b7a-0469c73e4c3b\") " pod="openshift-console/console-6db5565879-ndj8n" Oct 30 21:40:35.043053 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.041259 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:35.043053 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.041309 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-multus-daemon-config\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:40:35.043053 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.041529 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b8ed5eda-cbec-4adf-93b1-0ee0f1203f30-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bhdkg\" (UID: \"b8ed5eda-cbec-4adf-93b1-0ee0f1203f30\") " pod="openshift-insights/insights-runtime-extractor-bhdkg" Oct 30 21:40:35.043053 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.041556 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8b0ede8c-1c7c-479c-8c04-29d849fdbf43-serviceca\") pod \"node-ca-m7qjt\" (UID: \"8b0ede8c-1c7c-479c-8c04-29d849fdbf43\") " pod="openshift-image-registry/node-ca-m7qjt" Oct 30 21:40:35.043053 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.041556 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/971a083e-69b8-46d1-b56a-9ccf0f4d557d-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-44hwx\" (UID: \"971a083e-69b8-46d1-b56a-9ccf0f4d557d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-44hwx" Oct 30 21:40:35.043053 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.041628 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:35.043053 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.041814 6720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 21:40:35.043053 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.042279 6720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 21:40:35.045487 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.045469 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/971a083e-69b8-46d1-b56a-9ccf0f4d557d-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-44hwx\" (UID: \"971a083e-69b8-46d1-b56a-9ccf0f4d557d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-44hwx" Oct 30 21:40:35.047671 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.047652 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlvnp\" (UniqueName: \"kubernetes.io/projected/e2c4958f-c7a5-4b6a-9b29-c7e2cd7d2d41-kube-api-access-zlvnp\") pod \"dns-default-v9jlm\" (UID: \"e2c4958f-c7a5-4b6a-9b29-c7e2cd7d2d41\") " pod="openshift-dns/dns-default-v9jlm" Oct 30 21:40:35.047926 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.047910 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tpzc\" (UniqueName: \"kubernetes.io/projected/df51787b-87eb-420c-aa69-3d2762bdd386-kube-api-access-5tpzc\") pod \"network-check-target-9h6ck\" (UID: \"df51787b-87eb-420c-aa69-3d2762bdd386\") " pod="openshift-network-diagnostics/network-check-target-9h6ck" Oct 30 21:40:35.048074 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.048056 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7x77\" (UniqueName: \"kubernetes.io/projected/3018f90d-07fd-4d10-80a3-1f14248e638c-kube-api-access-k7x77\") pod \"ingress-canary-tj676\" (UID: \"3018f90d-07fd-4d10-80a3-1f14248e638c\") " pod="openshift-ingress-canary/ingress-canary-tj676" Oct 30 21:40:35.048133 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:35.048117 6720 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Oct 30 21:40:35.048172 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:35.048139 6720 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Oct 30 21:40:35.048172 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:35.048154 6720 projected.go:196] Error preparing data for projected volume kube-api-access-rczmc for pod openshift-console/console-6db5565879-ndj8n: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 21:40:35.048234 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:35.048223 6720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ec3fe241-c753-4845-8b7a-0469c73e4c3b-kube-api-access-rczmc podName:ec3fe241-c753-4845-8b7a-0469c73e4c3b nodeName:}" failed. No retries permitted until 2026-10-30 21:40:35.548208428 +0000 UTC m=+1.719173619 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-rczmc" (UniqueName: "kubernetes.io/projected/ec3fe241-c753-4845-8b7a-0469c73e4c3b-kube-api-access-rczmc") pod "console-6db5565879-ndj8n" (UID: "ec3fe241-c753-4845-8b7a-0469c73e4c3b") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 21:40:35.048234 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.048222 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-284h6\" (UniqueName: \"kubernetes.io/projected/7d451a16-523d-4b43-95cc-de9eeffd8402-kube-api-access-284h6\") pod \"multus-additional-cni-plugins-j8vz9\" (UID: \"7d451a16-523d-4b43-95cc-de9eeffd8402\") " pod="openshift-multus/multus-additional-cni-plugins-j8vz9" Oct 30 21:40:35.048560 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.048540 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:35.048852 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.048832 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm68d\" (UniqueName: \"kubernetes.io/projected/b8ed5eda-cbec-4adf-93b1-0ee0f1203f30-kube-api-access-gm68d\") pod \"insights-runtime-extractor-bhdkg\" (UID: \"b8ed5eda-cbec-4adf-93b1-0ee0f1203f30\") " pod="openshift-insights/insights-runtime-extractor-bhdkg" Oct 30 21:40:35.048922 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.048907 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj7w5\" (UniqueName: \"kubernetes.io/projected/d0a1535a-33cb-4a99-899d-aa8b3333326f-kube-api-access-sj7w5\") pod \"iptables-alerter-bzhn4\" (UID: \"d0a1535a-33cb-4a99-899d-aa8b3333326f\") " pod="openshift-network-operator/iptables-alerter-bzhn4" Oct 30 21:40:35.048977 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.048940 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvrrj\" (UniqueName: \"kubernetes.io/projected/df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d-kube-api-access-bvrrj\") pod \"metrics-server-c74454f78-xph4m\" (UID: \"df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d\") " pod="openshift-monitoring/metrics-server-c74454f78-xph4m" Oct 30 21:40:35.049363 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.049347 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2kzt\" (UniqueName: \"kubernetes.io/projected/970cd7bd-9018-4f3c-976a-af47bab1460a-kube-api-access-b2kzt\") pod \"tuned-vsgzd\" (UID: \"970cd7bd-9018-4f3c-976a-af47bab1460a\") " pod="openshift-cluster-node-tuning-operator/tuned-vsgzd" Oct 30 21:40:35.050294 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.050278 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:35.051749 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.051730 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l725d\" (UniqueName: \"kubernetes.io/projected/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-kube-api-access-l725d\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:40:35.051815 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.051797 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qchq\" (UniqueName: \"kubernetes.io/projected/90136025-abf7-4dc6-ac4b-c3050fc417f6-kube-api-access-6qchq\") pod \"thanos-querier-6fd8fdc4c5-xlslr\" (UID: \"90136025-abf7-4dc6-ac4b-c3050fc417f6\") " pod="openshift-monitoring/thanos-querier-6fd8fdc4c5-xlslr" Oct 30 21:40:35.056537 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.056519 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9874c\" (UniqueName: \"kubernetes.io/projected/a2e2c1af-8ab2-49c6-91ac-4712b88e0571-kube-api-access-9874c\") pod \"monitoring-plugin-77f4f76594-qv74l\" (UID: \"a2e2c1af-8ab2-49c6-91ac-4712b88e0571\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-qv74l" Oct 30 21:40:35.056636 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.056615 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c85rm\" (UniqueName: \"kubernetes.io/projected/64995343-c3c1-4915-953a-d1e4a5252461-kube-api-access-c85rm\") pod \"aws-ebs-csi-driver-node-kf9nx\" (UID: \"64995343-c3c1-4915-953a-d1e4a5252461\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kf9nx" Oct 30 21:40:35.056731 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.056716 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wv4r\" (UniqueName: \"kubernetes.io/projected/568ba29e-ccc3-4a4c-9a4f-192c0b931b0d-kube-api-access-5wv4r\") pod \"prometheus-operator-9cd89b74c-hrkzk\" (UID: \"568ba29e-ccc3-4a4c-9a4f-192c0b931b0d\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-hrkzk" Oct 30 21:40:35.056943 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.056923 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ldlh\" (UniqueName: \"kubernetes.io/projected/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-kube-api-access-4ldlh\") pod \"prometheus-k8s-0\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:35.061767 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.061743 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgrp4\" (UniqueName: \"kubernetes.io/projected/77670d00-a9cb-4607-bfb3-61ae91253cca-kube-api-access-mgrp4\") pod \"node-resolver-pnfw6\" (UID: \"77670d00-a9cb-4607-bfb3-61ae91253cca\") " pod="openshift-dns/node-resolver-pnfw6" Oct 30 21:40:35.061909 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.061753 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbcgc\" (UniqueName: \"kubernetes.io/projected/0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4-kube-api-access-pbcgc\") pod \"network-metrics-daemon-68gsj\" (UID: \"0823b66f-7f92-4bd2-9dc9-7d8aa6f03fe4\") " pod="openshift-multus/network-metrics-daemon-68gsj" Oct 30 21:40:35.061909 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.061785 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpmtf\" (UniqueName: \"kubernetes.io/projected/c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62-kube-api-access-qpmtf\") pod \"node-exporter-hbg8k\" (UID: \"c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62\") " pod="openshift-monitoring/node-exporter-hbg8k" Oct 30 21:40:35.061909 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.061824 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sl5w\" (UniqueName: \"kubernetes.io/projected/8b0ede8c-1c7c-479c-8c04-29d849fdbf43-kube-api-access-6sl5w\") pod \"node-ca-m7qjt\" (UID: \"8b0ede8c-1c7c-479c-8c04-29d849fdbf43\") " pod="openshift-image-registry/node-ca-m7qjt" Oct 30 21:40:35.061909 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.061872 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2stzq\" (UniqueName: \"kubernetes.io/projected/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-kube-api-access-2stzq\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:40:35.062150 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.061988 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfw8c\" (UniqueName: \"kubernetes.io/projected/971a083e-69b8-46d1-b56a-9ccf0f4d557d-kube-api-access-vfw8c\") pod \"openshift-state-metrics-7bd7489957-44hwx\" (UID: \"971a083e-69b8-46d1-b56a-9ccf0f4d557d\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-44hwx" Oct 30 21:40:35.062376 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.062356 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgq7t\" (UniqueName: \"kubernetes.io/projected/f4b104cb-8d0c-4519-86e0-07ea0e71a4a0-kube-api-access-wgq7t\") pod \"kube-state-metrics-5dc8d7d89b-4xbzf\" (UID: \"f4b104cb-8d0c-4519-86e0-07ea0e71a4a0\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-4xbzf" Oct 30 21:40:35.142123 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.142096 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-host-run-multus-certs\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:40:35.142248 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.142130 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-etc-openvswitch\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:40:35.142248 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.142152 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-host-run-ovn-kubernetes\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:40:35.142248 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.142174 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-etc-kubernetes\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:40:35.142248 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.142194 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-host-run-multus-certs\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:40:35.142248 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.142201 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-system-cni-dir\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:40:35.142248 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.142217 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-etc-openvswitch\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:40:35.142248 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.142217 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-host-run-ovn-kubernetes\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:40:35.142248 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.142243 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-cnibin\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:40:35.142671 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.142247 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-etc-kubernetes\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:40:35.142671 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.142253 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-system-cni-dir\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:40:35.142671 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.142282 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-cnibin\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:40:35.142671 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.142300 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-systemd-units\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:40:35.142671 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.142326 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-node-log\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:40:35.142671 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.142350 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d0a1535a-33cb-4a99-899d-aa8b3333326f-host-slash\") pod \"iptables-alerter-bzhn4\" (UID: \"d0a1535a-33cb-4a99-899d-aa8b3333326f\") " pod="openshift-network-operator/iptables-alerter-bzhn4" Oct 30 21:40:35.142671 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.142369 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-run-openvswitch\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:40:35.142671 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.142411 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-node-log\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:40:35.142671 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.142419 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-run-systemd\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:40:35.142671 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.142429 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-systemd-units\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:40:35.142671 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.142444 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/970cd7bd-9018-4f3c-976a-af47bab1460a-var-lib-kubelet\") pod \"tuned-vsgzd\" (UID: \"970cd7bd-9018-4f3c-976a-af47bab1460a\") " pod="openshift-cluster-node-tuning-operator/tuned-vsgzd" Oct 30 21:40:35.142671 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.142445 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d0a1535a-33cb-4a99-899d-aa8b3333326f-host-slash\") pod \"iptables-alerter-bzhn4\" (UID: \"d0a1535a-33cb-4a99-899d-aa8b3333326f\") " pod="openshift-network-operator/iptables-alerter-bzhn4" Oct 30 21:40:35.142671 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.142459 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-run-openvswitch\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:40:35.142671 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.142462 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-run-systemd\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:40:35.142671 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.142479 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/64995343-c3c1-4915-953a-d1e4a5252461-registration-dir\") pod \"aws-ebs-csi-driver-node-kf9nx\" (UID: \"64995343-c3c1-4915-953a-d1e4a5252461\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kf9nx" Oct 30 21:40:35.142671 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.142510 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-host-var-lib-kubelet\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:40:35.142671 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.142522 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/64995343-c3c1-4915-953a-d1e4a5252461-registration-dir\") pod \"aws-ebs-csi-driver-node-kf9nx\" (UID: \"64995343-c3c1-4915-953a-d1e4a5252461\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kf9nx" Oct 30 21:40:35.142671 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.142521 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/970cd7bd-9018-4f3c-976a-af47bab1460a-var-lib-kubelet\") pod \"tuned-vsgzd\" (UID: \"970cd7bd-9018-4f3c-976a-af47bab1460a\") " pod="openshift-cluster-node-tuning-operator/tuned-vsgzd" Oct 30 21:40:35.142671 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.142534 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/970cd7bd-9018-4f3c-976a-af47bab1460a-run\") pod \"tuned-vsgzd\" (UID: \"970cd7bd-9018-4f3c-976a-af47bab1460a\") " pod="openshift-cluster-node-tuning-operator/tuned-vsgzd" Oct 30 21:40:35.142671 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.142544 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-host-var-lib-kubelet\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:40:35.142671 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.142563 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:40:35.142671 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.142572 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/970cd7bd-9018-4f3c-976a-af47bab1460a-run\") pod \"tuned-vsgzd\" (UID: \"970cd7bd-9018-4f3c-976a-af47bab1460a\") " pod="openshift-cluster-node-tuning-operator/tuned-vsgzd" Oct 30 21:40:35.142671 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.142575 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:40:35.142671 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.142591 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/970cd7bd-9018-4f3c-976a-af47bab1460a-etc-sysctl-d\") pod \"tuned-vsgzd\" (UID: \"970cd7bd-9018-4f3c-976a-af47bab1460a\") " pod="openshift-cluster-node-tuning-operator/tuned-vsgzd" Oct 30 21:40:35.142671 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.142616 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-host-var-lib-cni-multus\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:40:35.142671 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.142638 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-multus-conf-dir\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:40:35.142671 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.142661 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/970cd7bd-9018-4f3c-976a-af47bab1460a-etc-sysconfig\") pod \"tuned-vsgzd\" (UID: \"970cd7bd-9018-4f3c-976a-af47bab1460a\") " pod="openshift-cluster-node-tuning-operator/tuned-vsgzd" Oct 30 21:40:35.142671 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.142689 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-multus-cni-dir\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:40:35.143805 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.142696 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-host-var-lib-cni-multus\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:40:35.143805 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.142704 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-multus-conf-dir\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:40:35.143805 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.142718 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/64995343-c3c1-4915-953a-d1e4a5252461-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kf9nx\" (UID: \"64995343-c3c1-4915-953a-d1e4a5252461\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kf9nx" Oct 30 21:40:35.143805 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.142725 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/970cd7bd-9018-4f3c-976a-af47bab1460a-etc-sysctl-d\") pod \"tuned-vsgzd\" (UID: \"970cd7bd-9018-4f3c-976a-af47bab1460a\") " pod="openshift-cluster-node-tuning-operator/tuned-vsgzd" Oct 30 21:40:35.143805 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.142728 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/970cd7bd-9018-4f3c-976a-af47bab1460a-etc-sysconfig\") pod \"tuned-vsgzd\" (UID: \"970cd7bd-9018-4f3c-976a-af47bab1460a\") " pod="openshift-cluster-node-tuning-operator/tuned-vsgzd" Oct 30 21:40:35.143805 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.142749 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-run-ovn\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:40:35.143805 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.142772 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/64995343-c3c1-4915-953a-d1e4a5252461-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kf9nx\" (UID: \"64995343-c3c1-4915-953a-d1e4a5252461\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kf9nx" Oct 30 21:40:35.143805 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.142773 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-host-cni-bin\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:40:35.143805 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.142792 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-run-ovn\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:40:35.143805 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.142802 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-multus-cni-dir\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:40:35.143805 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.142815 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/970cd7bd-9018-4f3c-976a-af47bab1460a-etc-kubernetes\") pod \"tuned-vsgzd\" (UID: \"970cd7bd-9018-4f3c-976a-af47bab1460a\") " pod="openshift-cluster-node-tuning-operator/tuned-vsgzd" Oct 30 21:40:35.143805 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.142829 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-host-cni-bin\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:40:35.143805 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.142845 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-multus-socket-dir-parent\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:40:35.143805 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.142870 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/970cd7bd-9018-4f3c-976a-af47bab1460a-etc-kubernetes\") pod \"tuned-vsgzd\" (UID: \"970cd7bd-9018-4f3c-976a-af47bab1460a\") " pod="openshift-cluster-node-tuning-operator/tuned-vsgzd" Oct 30 21:40:35.143805 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.142876 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-host-run-k8s-cni-cncf-io\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:40:35.143805 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.142899 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-host-run-netns\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:40:35.143805 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.142906 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-multus-socket-dir-parent\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:40:35.143805 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.142922 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7d451a16-523d-4b43-95cc-de9eeffd8402-cnibin\") pod \"multus-additional-cni-plugins-j8vz9\" (UID: \"7d451a16-523d-4b43-95cc-de9eeffd8402\") " pod="openshift-multus/multus-additional-cni-plugins-j8vz9" Oct 30 21:40:35.143805 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.142945 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7d451a16-523d-4b43-95cc-de9eeffd8402-tuning-conf-dir\") pod \"multus-additional-cni-plugins-j8vz9\" (UID: \"7d451a16-523d-4b43-95cc-de9eeffd8402\") " pod="openshift-multus/multus-additional-cni-plugins-j8vz9" Oct 30 21:40:35.143805 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.142945 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-host-run-k8s-cni-cncf-io\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:40:35.143805 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.142962 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-host-run-netns\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:40:35.143805 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.142982 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-host-slash\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:40:35.143805 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.142992 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7d451a16-523d-4b43-95cc-de9eeffd8402-cnibin\") pod \"multus-additional-cni-plugins-j8vz9\" (UID: \"7d451a16-523d-4b43-95cc-de9eeffd8402\") " pod="openshift-multus/multus-additional-cni-plugins-j8vz9" Oct 30 21:40:35.143805 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.143007 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-log-socket\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:40:35.143805 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.143012 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7d451a16-523d-4b43-95cc-de9eeffd8402-tuning-conf-dir\") pod \"multus-additional-cni-plugins-j8vz9\" (UID: \"7d451a16-523d-4b43-95cc-de9eeffd8402\") " pod="openshift-multus/multus-additional-cni-plugins-j8vz9" Oct 30 21:40:35.143805 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.143027 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-host-slash\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:40:35.143805 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.143034 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/970cd7bd-9018-4f3c-976a-af47bab1460a-etc-sysctl-conf\") pod \"tuned-vsgzd\" (UID: \"970cd7bd-9018-4f3c-976a-af47bab1460a\") " pod="openshift-cluster-node-tuning-operator/tuned-vsgzd" Oct 30 21:40:35.143805 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.143055 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62-sys\") pod \"node-exporter-hbg8k\" (UID: \"c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62\") " pod="openshift-monitoring/node-exporter-hbg8k" Oct 30 21:40:35.143805 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.143071 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/77670d00-a9cb-4607-bfb3-61ae91253cca-hosts-file\") pod \"node-resolver-pnfw6\" (UID: \"77670d00-a9cb-4607-bfb3-61ae91253cca\") " pod="openshift-dns/node-resolver-pnfw6" Oct 30 21:40:35.143805 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.143054 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-log-socket\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:40:35.143805 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.143087 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7d451a16-523d-4b43-95cc-de9eeffd8402-system-cni-dir\") pod \"multus-additional-cni-plugins-j8vz9\" (UID: \"7d451a16-523d-4b43-95cc-de9eeffd8402\") " pod="openshift-multus/multus-additional-cni-plugins-j8vz9" Oct 30 21:40:35.143805 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.143106 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62-sys\") pod \"node-exporter-hbg8k\" (UID: \"c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62\") " pod="openshift-monitoring/node-exporter-hbg8k" Oct 30 21:40:35.143805 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.143139 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/77670d00-a9cb-4607-bfb3-61ae91253cca-hosts-file\") pod \"node-resolver-pnfw6\" (UID: \"77670d00-a9cb-4607-bfb3-61ae91253cca\") " pod="openshift-dns/node-resolver-pnfw6" Oct 30 21:40:35.143805 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.143142 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/970cd7bd-9018-4f3c-976a-af47bab1460a-lib-modules\") pod \"tuned-vsgzd\" (UID: \"970cd7bd-9018-4f3c-976a-af47bab1460a\") " pod="openshift-cluster-node-tuning-operator/tuned-vsgzd" Oct 30 21:40:35.143805 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.143160 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/64995343-c3c1-4915-953a-d1e4a5252461-device-dir\") pod \"aws-ebs-csi-driver-node-kf9nx\" (UID: \"64995343-c3c1-4915-953a-d1e4a5252461\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kf9nx" Oct 30 21:40:35.143805 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.143187 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62-node-exporter-wtmp\") pod \"node-exporter-hbg8k\" (UID: \"c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62\") " pod="openshift-monitoring/node-exporter-hbg8k" Oct 30 21:40:35.143805 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.143196 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7d451a16-523d-4b43-95cc-de9eeffd8402-system-cni-dir\") pod \"multus-additional-cni-plugins-j8vz9\" (UID: \"7d451a16-523d-4b43-95cc-de9eeffd8402\") " pod="openshift-multus/multus-additional-cni-plugins-j8vz9" Oct 30 21:40:35.143805 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.143203 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-host-run-netns\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:40:35.143805 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.143237 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/64995343-c3c1-4915-953a-d1e4a5252461-device-dir\") pod \"aws-ebs-csi-driver-node-kf9nx\" (UID: \"64995343-c3c1-4915-953a-d1e4a5252461\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kf9nx" Oct 30 21:40:35.143805 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.143236 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-host-run-netns\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:40:35.143805 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.143250 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/970cd7bd-9018-4f3c-976a-af47bab1460a-etc-sysctl-conf\") pod \"tuned-vsgzd\" (UID: \"970cd7bd-9018-4f3c-976a-af47bab1460a\") " pod="openshift-cluster-node-tuning-operator/tuned-vsgzd" Oct 30 21:40:35.143805 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.143252 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-var-lib-openvswitch\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:40:35.143805 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.143275 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-var-lib-openvswitch\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:40:35.143805 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.143283 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62-root\") pod \"node-exporter-hbg8k\" (UID: \"c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62\") " pod="openshift-monitoring/node-exporter-hbg8k" Oct 30 21:40:35.143805 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.143283 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62-node-exporter-wtmp\") pod \"node-exporter-hbg8k\" (UID: \"c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62\") " pod="openshift-monitoring/node-exporter-hbg8k" Oct 30 21:40:35.143805 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.143299 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/970cd7bd-9018-4f3c-976a-af47bab1460a-etc-systemd\") pod \"tuned-vsgzd\" (UID: \"970cd7bd-9018-4f3c-976a-af47bab1460a\") " pod="openshift-cluster-node-tuning-operator/tuned-vsgzd" Oct 30 21:40:35.143805 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.143303 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62-root\") pod \"node-exporter-hbg8k\" (UID: \"c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62\") " pod="openshift-monitoring/node-exporter-hbg8k" Oct 30 21:40:35.143805 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.143257 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/970cd7bd-9018-4f3c-976a-af47bab1460a-lib-modules\") pod \"tuned-vsgzd\" (UID: \"970cd7bd-9018-4f3c-976a-af47bab1460a\") " pod="openshift-cluster-node-tuning-operator/tuned-vsgzd" Oct 30 21:40:35.143805 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.143340 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/970cd7bd-9018-4f3c-976a-af47bab1460a-etc-systemd\") pod \"tuned-vsgzd\" (UID: \"970cd7bd-9018-4f3c-976a-af47bab1460a\") " pod="openshift-cluster-node-tuning-operator/tuned-vsgzd" Oct 30 21:40:35.143805 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.143349 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/64995343-c3c1-4915-953a-d1e4a5252461-etc-selinux\") pod \"aws-ebs-csi-driver-node-kf9nx\" (UID: \"64995343-c3c1-4915-953a-d1e4a5252461\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kf9nx" Oct 30 21:40:35.143805 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.143369 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/17127db1-cf66-4951-ba5d-eb486ee9d3d5-kubelet-config\") pod \"global-pull-secret-syncer-vckqz\" (UID: \"17127db1-cf66-4951-ba5d-eb486ee9d3d5\") " pod="kube-system/global-pull-secret-syncer-vckqz" Oct 30 21:40:35.143805 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.143374 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/64995343-c3c1-4915-953a-d1e4a5252461-etc-selinux\") pod \"aws-ebs-csi-driver-node-kf9nx\" (UID: \"64995343-c3c1-4915-953a-d1e4a5252461\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kf9nx" Oct 30 21:40:35.143805 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.143403 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/17127db1-cf66-4951-ba5d-eb486ee9d3d5-dbus\") pod \"global-pull-secret-syncer-vckqz\" (UID: \"17127db1-cf66-4951-ba5d-eb486ee9d3d5\") " pod="kube-system/global-pull-secret-syncer-vckqz" Oct 30 21:40:35.143805 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.143419 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/64995343-c3c1-4915-953a-d1e4a5252461-socket-dir\") pod \"aws-ebs-csi-driver-node-kf9nx\" (UID: \"64995343-c3c1-4915-953a-d1e4a5252461\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kf9nx" Oct 30 21:40:35.143805 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.143428 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/17127db1-cf66-4951-ba5d-eb486ee9d3d5-kubelet-config\") pod \"global-pull-secret-syncer-vckqz\" (UID: \"17127db1-cf66-4951-ba5d-eb486ee9d3d5\") " pod="kube-system/global-pull-secret-syncer-vckqz" Oct 30 21:40:35.143805 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.143439 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-host-cni-netd\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:40:35.143805 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.143454 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b8ed5eda-cbec-4adf-93b1-0ee0f1203f30-crio-socket\") pod \"insights-runtime-extractor-bhdkg\" (UID: \"b8ed5eda-cbec-4adf-93b1-0ee0f1203f30\") " pod="openshift-insights/insights-runtime-extractor-bhdkg" Oct 30 21:40:35.143805 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.143460 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/64995343-c3c1-4915-953a-d1e4a5252461-socket-dir\") pod \"aws-ebs-csi-driver-node-kf9nx\" (UID: \"64995343-c3c1-4915-953a-d1e4a5252461\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kf9nx" Oct 30 21:40:35.143805 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.143468 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/970cd7bd-9018-4f3c-976a-af47bab1460a-etc-modprobe-d\") pod \"tuned-vsgzd\" (UID: \"970cd7bd-9018-4f3c-976a-af47bab1460a\") " pod="openshift-cluster-node-tuning-operator/tuned-vsgzd" Oct 30 21:40:35.143805 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.143466 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/17127db1-cf66-4951-ba5d-eb486ee9d3d5-dbus\") pod \"global-pull-secret-syncer-vckqz\" (UID: \"17127db1-cf66-4951-ba5d-eb486ee9d3d5\") " pod="kube-system/global-pull-secret-syncer-vckqz" Oct 30 21:40:35.143805 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.143481 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/970cd7bd-9018-4f3c-976a-af47bab1460a-sys\") pod \"tuned-vsgzd\" (UID: \"970cd7bd-9018-4f3c-976a-af47bab1460a\") " pod="openshift-cluster-node-tuning-operator/tuned-vsgzd" Oct 30 21:40:35.143805 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.143502 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/970cd7bd-9018-4f3c-976a-af47bab1460a-host\") pod \"tuned-vsgzd\" (UID: \"970cd7bd-9018-4f3c-976a-af47bab1460a\") " pod="openshift-cluster-node-tuning-operator/tuned-vsgzd" Oct 30 21:40:35.143805 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.143482 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-host-cni-netd\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:40:35.143805 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.143532 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/970cd7bd-9018-4f3c-976a-af47bab1460a-sys\") pod \"tuned-vsgzd\" (UID: \"970cd7bd-9018-4f3c-976a-af47bab1460a\") " pod="openshift-cluster-node-tuning-operator/tuned-vsgzd" Oct 30 21:40:35.143805 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.143519 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b8ed5eda-cbec-4adf-93b1-0ee0f1203f30-crio-socket\") pod \"insights-runtime-extractor-bhdkg\" (UID: \"b8ed5eda-cbec-4adf-93b1-0ee0f1203f30\") " pod="openshift-insights/insights-runtime-extractor-bhdkg" Oct 30 21:40:35.143805 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.143523 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-host-var-lib-cni-bin\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:40:35.143805 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.143552 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-host-var-lib-cni-bin\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:40:35.143805 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.143561 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8b0ede8c-1c7c-479c-8c04-29d849fdbf43-host\") pod \"node-ca-m7qjt\" (UID: \"8b0ede8c-1c7c-479c-8c04-29d849fdbf43\") " pod="openshift-image-registry/node-ca-m7qjt" Oct 30 21:40:35.143805 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.143574 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/970cd7bd-9018-4f3c-976a-af47bab1460a-etc-modprobe-d\") pod \"tuned-vsgzd\" (UID: \"970cd7bd-9018-4f3c-976a-af47bab1460a\") " pod="openshift-cluster-node-tuning-operator/tuned-vsgzd" Oct 30 21:40:35.143805 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.143580 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/970cd7bd-9018-4f3c-976a-af47bab1460a-host\") pod \"tuned-vsgzd\" (UID: \"970cd7bd-9018-4f3c-976a-af47bab1460a\") " pod="openshift-cluster-node-tuning-operator/tuned-vsgzd" Oct 30 21:40:35.143805 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.143586 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-host-kubelet\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:40:35.143805 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.143606 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/451ddc8c-9938-4c6a-a210-ff9a6efd7f70-host-kubelet\") pod \"ovnkube-node-75z8j\" (UID: \"451ddc8c-9938-4c6a-a210-ff9a6efd7f70\") " pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:40:35.143805 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.143617 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/64995343-c3c1-4915-953a-d1e4a5252461-sys-fs\") pod \"aws-ebs-csi-driver-node-kf9nx\" (UID: \"64995343-c3c1-4915-953a-d1e4a5252461\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kf9nx" Oct 30 21:40:35.143805 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.143629 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8b0ede8c-1c7c-479c-8c04-29d849fdbf43-host\") pod \"node-ca-m7qjt\" (UID: \"8b0ede8c-1c7c-479c-8c04-29d849fdbf43\") " pod="openshift-image-registry/node-ca-m7qjt" Oct 30 21:40:35.143805 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.143644 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-hostroot\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:40:35.143805 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.143707 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/64995343-c3c1-4915-953a-d1e4a5252461-sys-fs\") pod \"aws-ebs-csi-driver-node-kf9nx\" (UID: \"64995343-c3c1-4915-953a-d1e4a5252461\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kf9nx" Oct 30 21:40:35.143805 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.143708 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab-hostroot\") pod \"multus-25nw6\" (UID: \"7ffba6fe-fae3-4f37-9070-ba3d1c39e4ab\") " pod="openshift-multus/multus-25nw6" Oct 30 21:40:35.312276 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.312213 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-v9jlm" Oct 30 21:40:35.547143 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.547111 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ec3fe241-c753-4845-8b7a-0469c73e4c3b-service-ca\") pod \"console-6db5565879-ndj8n\" (UID: \"ec3fe241-c753-4845-8b7a-0469c73e4c3b\") " pod="openshift-console/console-6db5565879-ndj8n" Oct 30 21:40:35.547318 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.547152 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ec3fe241-c753-4845-8b7a-0469c73e4c3b-console-config\") pod \"console-6db5565879-ndj8n\" (UID: \"ec3fe241-c753-4845-8b7a-0469c73e4c3b\") " pod="openshift-console/console-6db5565879-ndj8n" Oct 30 21:40:35.547318 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.547199 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ec3fe241-c753-4845-8b7a-0469c73e4c3b-oauth-serving-cert\") pod \"console-6db5565879-ndj8n\" (UID: \"ec3fe241-c753-4845-8b7a-0469c73e4c3b\") " pod="openshift-console/console-6db5565879-ndj8n" Oct 30 21:40:35.547318 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.547236 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ec3fe241-c753-4845-8b7a-0469c73e4c3b-console-serving-cert\") pod \"console-6db5565879-ndj8n\" (UID: \"ec3fe241-c753-4845-8b7a-0469c73e4c3b\") " pod="openshift-console/console-6db5565879-ndj8n" Oct 30 21:40:35.547318 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:35.547240 6720 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Oct 30 21:40:35.547318 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.547267 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ec3fe241-c753-4845-8b7a-0469c73e4c3b-console-oauth-config\") pod \"console-6db5565879-ndj8n\" (UID: \"ec3fe241-c753-4845-8b7a-0469c73e4c3b\") " pod="openshift-console/console-6db5565879-ndj8n" Oct 30 21:40:35.547318 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:35.547290 6720 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Oct 30 21:40:35.547318 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:35.547312 6720 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Oct 30 21:40:35.547619 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:35.547355 6720 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Oct 30 21:40:35.547619 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:35.547316 6720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ec3fe241-c753-4845-8b7a-0469c73e4c3b-service-ca podName:ec3fe241-c753-4845-8b7a-0469c73e4c3b nodeName:}" failed. No retries permitted until 2026-10-30 21:40:36.547298042 +0000 UTC m=+2.718263221 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/ec3fe241-c753-4845-8b7a-0469c73e4c3b-service-ca") pod "console-6db5565879-ndj8n" (UID: "ec3fe241-c753-4845-8b7a-0469c73e4c3b") : object "openshift-console"/"service-ca" not registered Oct 30 21:40:35.547619 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:35.547405 6720 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Oct 30 21:40:35.547619 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:35.547421 6720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ec3fe241-c753-4845-8b7a-0469c73e4c3b-console-config podName:ec3fe241-c753-4845-8b7a-0469c73e4c3b nodeName:}" failed. No retries permitted until 2026-10-30 21:40:36.54740315 +0000 UTC m=+2.718368324 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/ec3fe241-c753-4845-8b7a-0469c73e4c3b-console-config") pod "console-6db5565879-ndj8n" (UID: "ec3fe241-c753-4845-8b7a-0469c73e4c3b") : object "openshift-console"/"console-config" not registered Oct 30 21:40:35.547619 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:35.547443 6720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ec3fe241-c753-4845-8b7a-0469c73e4c3b-oauth-serving-cert podName:ec3fe241-c753-4845-8b7a-0469c73e4c3b nodeName:}" failed. No retries permitted until 2026-10-30 21:40:36.547434188 +0000 UTC m=+2.718399356 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/ec3fe241-c753-4845-8b7a-0469c73e4c3b-oauth-serving-cert") pod "console-6db5565879-ndj8n" (UID: "ec3fe241-c753-4845-8b7a-0469c73e4c3b") : object "openshift-console"/"oauth-serving-cert" not registered Oct 30 21:40:35.547619 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:35.547461 6720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec3fe241-c753-4845-8b7a-0469c73e4c3b-console-oauth-config podName:ec3fe241-c753-4845-8b7a-0469c73e4c3b nodeName:}" failed. No retries permitted until 2026-10-30 21:40:36.547452637 +0000 UTC m=+2.718417803 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/ec3fe241-c753-4845-8b7a-0469c73e4c3b-console-oauth-config") pod "console-6db5565879-ndj8n" (UID: "ec3fe241-c753-4845-8b7a-0469c73e4c3b") : object "openshift-console"/"console-oauth-config" not registered Oct 30 21:40:35.547619 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:35.547478 6720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec3fe241-c753-4845-8b7a-0469c73e4c3b-console-serving-cert podName:ec3fe241-c753-4845-8b7a-0469c73e4c3b nodeName:}" failed. No retries permitted until 2026-10-30 21:40:36.547468675 +0000 UTC m=+2.718433836 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/ec3fe241-c753-4845-8b7a-0469c73e4c3b-console-serving-cert") pod "console-6db5565879-ndj8n" (UID: "ec3fe241-c753-4845-8b7a-0469c73e4c3b") : object "openshift-console"/"console-serving-cert" not registered Oct 30 21:40:35.647793 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:35.647762 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rczmc\" (UniqueName: \"kubernetes.io/projected/ec3fe241-c753-4845-8b7a-0469c73e4c3b-kube-api-access-rczmc\") pod \"console-6db5565879-ndj8n\" (UID: \"ec3fe241-c753-4845-8b7a-0469c73e4c3b\") " pod="openshift-console/console-6db5565879-ndj8n" Oct 30 21:40:35.647954 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:35.647937 6720 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Oct 30 21:40:35.648036 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:35.647963 6720 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Oct 30 21:40:35.648036 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:35.647976 6720 projected.go:196] Error preparing data for projected volume kube-api-access-rczmc for pod openshift-console/console-6db5565879-ndj8n: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 21:40:35.648139 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:35.648038 6720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ec3fe241-c753-4845-8b7a-0469c73e4c3b-kube-api-access-rczmc podName:ec3fe241-c753-4845-8b7a-0469c73e4c3b nodeName:}" failed. No retries permitted until 2026-10-30 21:40:36.648019041 +0000 UTC m=+2.818984215 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-rczmc" (UniqueName: "kubernetes.io/projected/ec3fe241-c753-4845-8b7a-0469c73e4c3b-kube-api-access-rczmc") pod "console-6db5565879-ndj8n" (UID: "ec3fe241-c753-4845-8b7a-0469c73e4c3b") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 21:40:36.051028 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:36.050961 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:40:36.555169 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:36.555137 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ec3fe241-c753-4845-8b7a-0469c73e4c3b-console-oauth-config\") pod \"console-6db5565879-ndj8n\" (UID: \"ec3fe241-c753-4845-8b7a-0469c73e4c3b\") " pod="openshift-console/console-6db5565879-ndj8n" Oct 30 21:40:36.555346 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:36.555180 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ec3fe241-c753-4845-8b7a-0469c73e4c3b-service-ca\") pod \"console-6db5565879-ndj8n\" (UID: \"ec3fe241-c753-4845-8b7a-0469c73e4c3b\") " pod="openshift-console/console-6db5565879-ndj8n" Oct 30 21:40:36.555346 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:36.555200 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ec3fe241-c753-4845-8b7a-0469c73e4c3b-console-config\") pod \"console-6db5565879-ndj8n\" (UID: \"ec3fe241-c753-4845-8b7a-0469c73e4c3b\") " pod="openshift-console/console-6db5565879-ndj8n" Oct 30 21:40:36.555346 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:36.555245 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ec3fe241-c753-4845-8b7a-0469c73e4c3b-oauth-serving-cert\") pod \"console-6db5565879-ndj8n\" (UID: \"ec3fe241-c753-4845-8b7a-0469c73e4c3b\") " pod="openshift-console/console-6db5565879-ndj8n" Oct 30 21:40:36.555346 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:36.555271 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ec3fe241-c753-4845-8b7a-0469c73e4c3b-console-serving-cert\") pod \"console-6db5565879-ndj8n\" (UID: \"ec3fe241-c753-4845-8b7a-0469c73e4c3b\") " pod="openshift-console/console-6db5565879-ndj8n" Oct 30 21:40:36.555346 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:36.555279 6720 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Oct 30 21:40:36.555346 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:36.555323 6720 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Oct 30 21:40:36.555649 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:36.555352 6720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ec3fe241-c753-4845-8b7a-0469c73e4c3b-service-ca podName:ec3fe241-c753-4845-8b7a-0469c73e4c3b nodeName:}" failed. No retries permitted until 2026-10-30 21:40:38.555333433 +0000 UTC m=+4.726298598 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/ec3fe241-c753-4845-8b7a-0469c73e4c3b-service-ca") pod "console-6db5565879-ndj8n" (UID: "ec3fe241-c753-4845-8b7a-0469c73e4c3b") : object "openshift-console"/"service-ca" not registered Oct 30 21:40:36.555649 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:36.555351 6720 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Oct 30 21:40:36.555649 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:36.555373 6720 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Oct 30 21:40:36.555649 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:36.555412 6720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec3fe241-c753-4845-8b7a-0469c73e4c3b-console-oauth-config podName:ec3fe241-c753-4845-8b7a-0469c73e4c3b nodeName:}" failed. No retries permitted until 2026-10-30 21:40:38.555376009 +0000 UTC m=+4.726341185 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/ec3fe241-c753-4845-8b7a-0469c73e4c3b-console-oauth-config") pod "console-6db5565879-ndj8n" (UID: "ec3fe241-c753-4845-8b7a-0469c73e4c3b") : object "openshift-console"/"console-oauth-config" not registered Oct 30 21:40:36.555649 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:36.555434 6720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ec3fe241-c753-4845-8b7a-0469c73e4c3b-oauth-serving-cert podName:ec3fe241-c753-4845-8b7a-0469c73e4c3b nodeName:}" failed. No retries permitted until 2026-10-30 21:40:38.555421495 +0000 UTC m=+4.726386677 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/ec3fe241-c753-4845-8b7a-0469c73e4c3b-oauth-serving-cert") pod "console-6db5565879-ndj8n" (UID: "ec3fe241-c753-4845-8b7a-0469c73e4c3b") : object "openshift-console"/"oauth-serving-cert" not registered Oct 30 21:40:36.555649 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:36.555452 6720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec3fe241-c753-4845-8b7a-0469c73e4c3b-console-serving-cert podName:ec3fe241-c753-4845-8b7a-0469c73e4c3b nodeName:}" failed. No retries permitted until 2026-10-30 21:40:38.555443886 +0000 UTC m=+4.726409051 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/ec3fe241-c753-4845-8b7a-0469c73e4c3b-console-serving-cert") pod "console-6db5565879-ndj8n" (UID: "ec3fe241-c753-4845-8b7a-0469c73e4c3b") : object "openshift-console"/"console-serving-cert" not registered Oct 30 21:40:36.555649 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:36.555478 6720 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Oct 30 21:40:36.555649 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:36.555510 6720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ec3fe241-c753-4845-8b7a-0469c73e4c3b-console-config podName:ec3fe241-c753-4845-8b7a-0469c73e4c3b nodeName:}" failed. No retries permitted until 2026-10-30 21:40:38.555502556 +0000 UTC m=+4.726467716 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/ec3fe241-c753-4845-8b7a-0469c73e4c3b-console-config") pod "console-6db5565879-ndj8n" (UID: "ec3fe241-c753-4845-8b7a-0469c73e4c3b") : object "openshift-console"/"console-config" not registered Oct 30 21:40:36.656550 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:36.656512 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rczmc\" (UniqueName: \"kubernetes.io/projected/ec3fe241-c753-4845-8b7a-0469c73e4c3b-kube-api-access-rczmc\") pod \"console-6db5565879-ndj8n\" (UID: \"ec3fe241-c753-4845-8b7a-0469c73e4c3b\") " pod="openshift-console/console-6db5565879-ndj8n" Oct 30 21:40:36.656704 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:36.656664 6720 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Oct 30 21:40:36.656704 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:36.656684 6720 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Oct 30 21:40:36.656704 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:36.656694 6720 projected.go:196] Error preparing data for projected volume kube-api-access-rczmc for pod openshift-console/console-6db5565879-ndj8n: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 21:40:36.656837 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:36.656739 6720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ec3fe241-c753-4845-8b7a-0469c73e4c3b-kube-api-access-rczmc podName:ec3fe241-c753-4845-8b7a-0469c73e4c3b nodeName:}" failed. No retries permitted until 2026-10-30 21:40:38.656727151 +0000 UTC m=+4.827692311 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-rczmc" (UniqueName: "kubernetes.io/projected/ec3fe241-c753-4845-8b7a-0469c73e4c3b-kube-api-access-rczmc") pod "console-6db5565879-ndj8n" (UID: "ec3fe241-c753-4845-8b7a-0469c73e4c3b") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 21:40:38.076992 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:38.076964 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-qv74l" Oct 30 21:40:38.574849 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:38.574819 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-c74454f78-xph4m" Oct 30 21:40:38.576407 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:38.576372 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ec3fe241-c753-4845-8b7a-0469c73e4c3b-service-ca\") pod \"console-6db5565879-ndj8n\" (UID: \"ec3fe241-c753-4845-8b7a-0469c73e4c3b\") " pod="openshift-console/console-6db5565879-ndj8n" Oct 30 21:40:38.576503 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:38.576426 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ec3fe241-c753-4845-8b7a-0469c73e4c3b-console-config\") pod \"console-6db5565879-ndj8n\" (UID: \"ec3fe241-c753-4845-8b7a-0469c73e4c3b\") " pod="openshift-console/console-6db5565879-ndj8n" Oct 30 21:40:38.576503 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:38.576459 6720 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Oct 30 21:40:38.576503 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:38.576502 6720 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Oct 30 21:40:38.576655 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:38.576531 6720 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Oct 30 21:40:38.576655 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:38.576465 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ec3fe241-c753-4845-8b7a-0469c73e4c3b-oauth-serving-cert\") pod \"console-6db5565879-ndj8n\" (UID: \"ec3fe241-c753-4845-8b7a-0469c73e4c3b\") " pod="openshift-console/console-6db5565879-ndj8n" Oct 30 21:40:38.576655 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:38.576536 6720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ec3fe241-c753-4845-8b7a-0469c73e4c3b-service-ca podName:ec3fe241-c753-4845-8b7a-0469c73e4c3b nodeName:}" failed. No retries permitted until 2026-10-30 21:40:42.576519256 +0000 UTC m=+8.747484434 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/ec3fe241-c753-4845-8b7a-0469c73e4c3b-service-ca") pod "console-6db5565879-ndj8n" (UID: "ec3fe241-c753-4845-8b7a-0469c73e4c3b") : object "openshift-console"/"service-ca" not registered Oct 30 21:40:38.576655 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:38.576599 6720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ec3fe241-c753-4845-8b7a-0469c73e4c3b-oauth-serving-cert podName:ec3fe241-c753-4845-8b7a-0469c73e4c3b nodeName:}" failed. No retries permitted until 2026-10-30 21:40:42.576586312 +0000 UTC m=+8.747551471 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/ec3fe241-c753-4845-8b7a-0469c73e4c3b-oauth-serving-cert") pod "console-6db5565879-ndj8n" (UID: "ec3fe241-c753-4845-8b7a-0469c73e4c3b") : object "openshift-console"/"oauth-serving-cert" not registered Oct 30 21:40:38.576655 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:38.576612 6720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ec3fe241-c753-4845-8b7a-0469c73e4c3b-console-config podName:ec3fe241-c753-4845-8b7a-0469c73e4c3b nodeName:}" failed. No retries permitted until 2026-10-30 21:40:42.576605132 +0000 UTC m=+8.747570292 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/ec3fe241-c753-4845-8b7a-0469c73e4c3b-console-config") pod "console-6db5565879-ndj8n" (UID: "ec3fe241-c753-4845-8b7a-0469c73e4c3b") : object "openshift-console"/"console-config" not registered Oct 30 21:40:38.576655 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:38.576645 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ec3fe241-c753-4845-8b7a-0469c73e4c3b-console-serving-cert\") pod \"console-6db5565879-ndj8n\" (UID: \"ec3fe241-c753-4845-8b7a-0469c73e4c3b\") " pod="openshift-console/console-6db5565879-ndj8n" Oct 30 21:40:38.576929 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:38.576680 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ec3fe241-c753-4845-8b7a-0469c73e4c3b-console-oauth-config\") pod \"console-6db5565879-ndj8n\" (UID: \"ec3fe241-c753-4845-8b7a-0469c73e4c3b\") " pod="openshift-console/console-6db5565879-ndj8n" Oct 30 21:40:38.576929 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:38.576757 6720 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Oct 30 21:40:38.576929 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:38.576779 6720 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Oct 30 21:40:38.576929 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:38.576785 6720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec3fe241-c753-4845-8b7a-0469c73e4c3b-console-oauth-config podName:ec3fe241-c753-4845-8b7a-0469c73e4c3b nodeName:}" failed. No retries permitted until 2026-10-30 21:40:42.576778843 +0000 UTC m=+8.747744003 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/ec3fe241-c753-4845-8b7a-0469c73e4c3b-console-oauth-config") pod "console-6db5565879-ndj8n" (UID: "ec3fe241-c753-4845-8b7a-0469c73e4c3b") : object "openshift-console"/"console-oauth-config" not registered Oct 30 21:40:38.576929 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:38.576852 6720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec3fe241-c753-4845-8b7a-0469c73e4c3b-console-serving-cert podName:ec3fe241-c753-4845-8b7a-0469c73e4c3b nodeName:}" failed. No retries permitted until 2026-10-30 21:40:42.576836963 +0000 UTC m=+8.747802129 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/ec3fe241-c753-4845-8b7a-0469c73e4c3b-console-serving-cert") pod "console-6db5565879-ndj8n" (UID: "ec3fe241-c753-4845-8b7a-0469c73e4c3b") : object "openshift-console"/"console-serving-cert" not registered Oct 30 21:40:38.677431 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:38.677372 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rczmc\" (UniqueName: \"kubernetes.io/projected/ec3fe241-c753-4845-8b7a-0469c73e4c3b-kube-api-access-rczmc\") pod \"console-6db5565879-ndj8n\" (UID: \"ec3fe241-c753-4845-8b7a-0469c73e4c3b\") " pod="openshift-console/console-6db5565879-ndj8n" Oct 30 21:40:38.677619 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:38.677549 6720 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Oct 30 21:40:38.677619 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:38.677574 6720 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Oct 30 21:40:38.677619 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:38.677588 6720 projected.go:196] Error preparing data for projected volume kube-api-access-rczmc for pod openshift-console/console-6db5565879-ndj8n: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 21:40:38.677784 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:38.677644 6720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ec3fe241-c753-4845-8b7a-0469c73e4c3b-kube-api-access-rczmc podName:ec3fe241-c753-4845-8b7a-0469c73e4c3b nodeName:}" failed. No retries permitted until 2026-10-30 21:40:42.677625797 +0000 UTC m=+8.848590977 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-rczmc" (UniqueName: "kubernetes.io/projected/ec3fe241-c753-4845-8b7a-0469c73e4c3b-kube-api-access-rczmc") pod "console-6db5565879-ndj8n" (UID: "ec3fe241-c753-4845-8b7a-0469c73e4c3b") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 21:40:39.142923 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:39.142891 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6fd8fdc4c5-xlslr" Oct 30 21:40:39.422725 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:39.422629 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-9h6ck" Oct 30 21:40:40.178066 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:40.178036 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:40.401510 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:40.401466 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 21:40:42.618556 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:42.618520 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ec3fe241-c753-4845-8b7a-0469c73e4c3b-console-oauth-config\") pod \"console-6db5565879-ndj8n\" (UID: \"ec3fe241-c753-4845-8b7a-0469c73e4c3b\") " pod="openshift-console/console-6db5565879-ndj8n" Oct 30 21:40:42.618963 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:42.618586 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ec3fe241-c753-4845-8b7a-0469c73e4c3b-service-ca\") pod \"console-6db5565879-ndj8n\" (UID: \"ec3fe241-c753-4845-8b7a-0469c73e4c3b\") " pod="openshift-console/console-6db5565879-ndj8n" Oct 30 21:40:42.618963 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:42.618618 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ec3fe241-c753-4845-8b7a-0469c73e4c3b-console-config\") pod \"console-6db5565879-ndj8n\" (UID: \"ec3fe241-c753-4845-8b7a-0469c73e4c3b\") " pod="openshift-console/console-6db5565879-ndj8n" Oct 30 21:40:42.618963 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:42.618673 6720 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Oct 30 21:40:42.618963 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:42.618717 6720 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Oct 30 21:40:42.618963 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:42.618723 6720 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Oct 30 21:40:42.618963 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:42.618746 6720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec3fe241-c753-4845-8b7a-0469c73e4c3b-console-oauth-config podName:ec3fe241-c753-4845-8b7a-0469c73e4c3b nodeName:}" failed. No retries permitted until 2026-10-30 21:40:50.618726745 +0000 UTC m=+16.789691906 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/ec3fe241-c753-4845-8b7a-0469c73e4c3b-console-oauth-config") pod "console-6db5565879-ndj8n" (UID: "ec3fe241-c753-4845-8b7a-0469c73e4c3b") : object "openshift-console"/"console-oauth-config" not registered Oct 30 21:40:42.618963 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:42.618679 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ec3fe241-c753-4845-8b7a-0469c73e4c3b-oauth-serving-cert\") pod \"console-6db5565879-ndj8n\" (UID: \"ec3fe241-c753-4845-8b7a-0469c73e4c3b\") " pod="openshift-console/console-6db5565879-ndj8n" Oct 30 21:40:42.618963 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:42.618771 6720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ec3fe241-c753-4845-8b7a-0469c73e4c3b-oauth-serving-cert podName:ec3fe241-c753-4845-8b7a-0469c73e4c3b nodeName:}" failed. No retries permitted until 2026-10-30 21:40:50.618755479 +0000 UTC m=+16.789720645 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/ec3fe241-c753-4845-8b7a-0469c73e4c3b-oauth-serving-cert") pod "console-6db5565879-ndj8n" (UID: "ec3fe241-c753-4845-8b7a-0469c73e4c3b") : object "openshift-console"/"oauth-serving-cert" not registered Oct 30 21:40:42.618963 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:42.618784 6720 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Oct 30 21:40:42.618963 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:42.618804 6720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ec3fe241-c753-4845-8b7a-0469c73e4c3b-service-ca podName:ec3fe241-c753-4845-8b7a-0469c73e4c3b nodeName:}" failed. No retries permitted until 2026-10-30 21:40:50.618786636 +0000 UTC m=+16.789751805 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/ec3fe241-c753-4845-8b7a-0469c73e4c3b-service-ca") pod "console-6db5565879-ndj8n" (UID: "ec3fe241-c753-4845-8b7a-0469c73e4c3b") : object "openshift-console"/"service-ca" not registered Oct 30 21:40:42.618963 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:42.618833 6720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ec3fe241-c753-4845-8b7a-0469c73e4c3b-console-config podName:ec3fe241-c753-4845-8b7a-0469c73e4c3b nodeName:}" failed. No retries permitted until 2026-10-30 21:40:50.618822 +0000 UTC m=+16.789787163 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/ec3fe241-c753-4845-8b7a-0469c73e4c3b-console-config") pod "console-6db5565879-ndj8n" (UID: "ec3fe241-c753-4845-8b7a-0469c73e4c3b") : object "openshift-console"/"console-config" not registered Oct 30 21:40:42.618963 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:42.618865 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ec3fe241-c753-4845-8b7a-0469c73e4c3b-console-serving-cert\") pod \"console-6db5565879-ndj8n\" (UID: \"ec3fe241-c753-4845-8b7a-0469c73e4c3b\") " pod="openshift-console/console-6db5565879-ndj8n" Oct 30 21:40:42.618963 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:42.618959 6720 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Oct 30 21:40:42.619442 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:42.618992 6720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec3fe241-c753-4845-8b7a-0469c73e4c3b-console-serving-cert podName:ec3fe241-c753-4845-8b7a-0469c73e4c3b nodeName:}" failed. No retries permitted until 2026-10-30 21:40:50.618984959 +0000 UTC m=+16.789950123 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/ec3fe241-c753-4845-8b7a-0469c73e4c3b-console-serving-cert") pod "console-6db5565879-ndj8n" (UID: "ec3fe241-c753-4845-8b7a-0469c73e4c3b") : object "openshift-console"/"console-serving-cert" not registered Oct 30 21:40:42.720084 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:42.720054 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rczmc\" (UniqueName: \"kubernetes.io/projected/ec3fe241-c753-4845-8b7a-0469c73e4c3b-kube-api-access-rczmc\") pod \"console-6db5565879-ndj8n\" (UID: \"ec3fe241-c753-4845-8b7a-0469c73e4c3b\") " pod="openshift-console/console-6db5565879-ndj8n" Oct 30 21:40:42.720227 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:42.720207 6720 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Oct 30 21:40:42.720227 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:42.720223 6720 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Oct 30 21:40:42.720299 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:42.720233 6720 projected.go:196] Error preparing data for projected volume kube-api-access-rczmc for pod openshift-console/console-6db5565879-ndj8n: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 21:40:42.720299 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:42.720281 6720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ec3fe241-c753-4845-8b7a-0469c73e4c3b-kube-api-access-rczmc podName:ec3fe241-c753-4845-8b7a-0469c73e4c3b nodeName:}" failed. No retries permitted until 2026-10-30 21:40:50.720268325 +0000 UTC m=+16.891233528 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-rczmc" (UniqueName: "kubernetes.io/projected/ec3fe241-c753-4845-8b7a-0469c73e4c3b-kube-api-access-rczmc") pod "console-6db5565879-ndj8n" (UID: "ec3fe241-c753-4845-8b7a-0469c73e4c3b") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 21:40:42.923428 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:42.923044 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:43.059188 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:43.059135 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:44.170240 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:44.170164 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:40:44.170638 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:44.170478 6720 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:40:45.407941 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:45.407913 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:45.550107 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:45.550076 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:48.270605 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:48.270567 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 21:40:48.271470 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:48.271416 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b" containerName="prometheus" containerID="cri-o://4b2bd085af3f718bc596d63c5ad8ded90ba183b796157a3432e67072c75467f7" gracePeriod=600 Oct 30 21:40:48.271702 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:48.271433 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b" containerName="kube-rbac-proxy-thanos" containerID="cri-o://a557f1280585b29505458f1f9fd878dceae79678a4f48ed21efc6d4c08bb82d8" gracePeriod=600 Oct 30 21:40:48.271874 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:48.271450 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b" containerName="kube-rbac-proxy" containerID="cri-o://dd38eb45d35ddcf72610143f96e71fc7ee928e71fd3f8eeda09afc20bcc49e26" gracePeriod=600 Oct 30 21:40:48.271874 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:48.271461 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b" containerName="kube-rbac-proxy-web" containerID="cri-o://fd0dbf8a054c508e3dbab8f958d4fa7655b02969db228af9671e7c1cc9f9a883" gracePeriod=600 Oct 30 21:40:48.271874 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:48.271480 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b" containerName="thanos-sidecar" containerID="cri-o://d894623a6390af62bd4043a10961f6e0c09665c6d5cf9e5694c540fca70213f4" gracePeriod=600 Oct 30 21:40:48.271874 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:48.271519 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b" containerName="config-reloader" containerID="cri-o://be222d32f0e90f6c8195bc3d07590a9be3e443b65fa6ff502ec81da31bacdf44" gracePeriod=600 Oct 30 21:40:48.533912 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:48.533887 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:48.681032 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:48.681000 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-tls-assets\" (UniqueName: \"kubernetes.io/projected/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-tls-assets\") pod \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " Oct 30 21:40:48.681213 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:48.681041 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-prometheus-k8s-rulefiles-2\") pod \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " Oct 30 21:40:48.681213 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:48.681061 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-secret-kube-rbac-proxy\") pod \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " Oct 30 21:40:48.681213 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:48.681081 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-configmap-serving-certs-ca-bundle\") pod \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " Oct 30 21:40:48.681213 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:48.681106 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-web-config\" (UniqueName: \"kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-web-config\") pod \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " Oct 30 21:40:48.681213 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:48.681149 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-prometheus-k8s-db\") pod \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " Oct 30 21:40:48.681213 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:48.681179 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-prometheus-trusted-ca-bundle\") pod \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " Oct 30 21:40:48.681213 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:48.681207 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " Oct 30 21:40:48.681606 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:48.681234 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-config\" (UniqueName: \"kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-config\") pod \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " Oct 30 21:40:48.681606 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:48.681277 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-secret-grpc-tls\") pod \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " Oct 30 21:40:48.681606 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:48.681326 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-configmap-metrics-client-ca\") pod \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " Oct 30 21:40:48.681606 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:48.681366 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-config-out\" (UniqueName: \"kubernetes.io/empty-dir/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-config-out\") pod \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " Oct 30 21:40:48.681606 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:48.681440 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-secret-metrics-client-certs\") pod \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " Oct 30 21:40:48.681606 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:48.681468 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-kube-api-access-4ldlh\" (UniqueName: \"kubernetes.io/projected/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-kube-api-access-4ldlh\") pod \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " Oct 30 21:40:48.681606 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:48.681477 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-prometheus-k8s-rulefiles-2" pod "1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b" (UID: "1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:40:48.681606 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:48.681507 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-configmap-kubelet-serving-ca-bundle\") pod \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " Oct 30 21:40:48.681606 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:48.681510 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-configmap-serving-certs-ca-bundle" pod "1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b" (UID: "1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:40:48.681606 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:48.681535 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-thanos-prometheus-http-client-file\") pod \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " Oct 30 21:40:48.681606 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:48.681583 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-prometheus-k8s-rulefiles-1\") pod \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " Oct 30 21:40:48.682318 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:48.681615 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-secret-prometheus-k8s-tls\") pod \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " Oct 30 21:40:48.682318 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:48.681663 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " Oct 30 21:40:48.682318 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:48.681720 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-prometheus-k8s-rulefiles-0\") pod \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\" (UID: \"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b\") " Oct 30 21:40:48.682318 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:48.682010 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-prometheus-trusted-ca-bundle" pod "1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b" (UID: "1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:40:48.682318 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:48.682061 6720 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:48.682318 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:48.682082 6720 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:48.682672 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:48.682644 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-prometheus-k8s-db" pod "1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b" (UID: "1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:40:48.683144 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:48.683113 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-configmap-metrics-client-ca" pod "1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b" (UID: "1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:40:48.683644 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:48.683454 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-prometheus-k8s-rulefiles-1" pod "1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b" (UID: "1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:40:48.684439 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:48.684417 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-configmap-kubelet-serving-ca-bundle" pod "1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b" (UID: "1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:40:48.685515 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:48.685481 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-tls-assets" pod "1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b" (UID: "1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:40:48.685680 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:48.685652 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-prometheus-k8s-rulefiles-0" pod "1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b" (UID: "1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:40:48.686021 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:48.686000 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-config-out" pod "1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b" (UID: "1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:40:48.686140 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:48.686049 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-secret-metrics-client-certs" pod "1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b" (UID: "1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:40:48.686237 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:48.686159 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-secret-kube-rbac-proxy" pod "1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b" (UID: "1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:40:48.686467 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:48.686445 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-config" pod "1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b" (UID: "1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:40:48.686660 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:48.686623 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-secret-grpc-tls" pod "1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b" (UID: "1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:40:48.686734 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:48.686683 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-kube-api-access-4ldlh" pod "1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b" (UID: "1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b"). InnerVolumeSpecName "kube-api-access-4ldlh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:40:48.686790 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:48.686747 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-secret-prometheus-k8s-tls" pod "1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b" (UID: "1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:40:48.686830 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:48.686812 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-thanos-prometheus-http-client-file" pod "1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b" (UID: "1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:40:48.686965 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:48.686946 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-secret-prometheus-k8s-kube-rbac-proxy-web" pod "1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b" (UID: "1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:40:48.687274 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:48.687257 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-secret-prometheus-k8s-thanos-sidecar-tls" pod "1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b" (UID: "1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:40:48.696153 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:48.696133 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-web-config" pod "1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b" (UID: "1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:40:48.783364 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:48.783306 6720 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:48.783364 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:48.783329 6720 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-tls-assets\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:48.783364 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:48.783340 6720 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-secret-kube-rbac-proxy\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:48.783364 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:48.783349 6720 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-web-config\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:48.783364 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:48.783358 6720 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-prometheus-k8s-db\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:48.783364 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:48.783367 6720 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-prometheus-trusted-ca-bundle\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:48.783638 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:48.783376 6720 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:48.783638 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:48.783404 6720 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-config\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:48.783638 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:48.783412 6720 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-secret-grpc-tls\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:48.783638 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:48.783420 6720 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-configmap-metrics-client-ca\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:48.783638 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:48.783428 6720 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-config-out\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:48.783638 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:48.783436 6720 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-secret-metrics-client-certs\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:48.783638 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:48.783444 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ldlh\" (UniqueName: \"kubernetes.io/projected/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-kube-api-access-4ldlh\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:48.783638 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:48.783453 6720 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:48.783638 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:48.783461 6720 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-thanos-prometheus-http-client-file\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:48.783638 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:48.783469 6720 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:48.783638 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:48.783478 6720 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-secret-prometheus-k8s-tls\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:48.783638 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:48.783486 6720 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:40:49.073329 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.073254 6720 generic.go:356] "Generic (PLEG): container finished" podID="1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b" containerID="a557f1280585b29505458f1f9fd878dceae79678a4f48ed21efc6d4c08bb82d8" exitCode=0 Oct 30 21:40:49.073329 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.073287 6720 generic.go:356] "Generic (PLEG): container finished" podID="1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b" containerID="dd38eb45d35ddcf72610143f96e71fc7ee928e71fd3f8eeda09afc20bcc49e26" exitCode=0 Oct 30 21:40:49.073329 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.073301 6720 generic.go:356] "Generic (PLEG): container finished" podID="1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b" containerID="fd0dbf8a054c508e3dbab8f958d4fa7655b02969db228af9671e7c1cc9f9a883" exitCode=0 Oct 30 21:40:49.073329 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.073310 6720 generic.go:356] "Generic (PLEG): container finished" podID="1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b" containerID="d894623a6390af62bd4043a10961f6e0c09665c6d5cf9e5694c540fca70213f4" exitCode=0 Oct 30 21:40:49.073329 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.073318 6720 generic.go:356] "Generic (PLEG): container finished" podID="1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b" containerID="be222d32f0e90f6c8195bc3d07590a9be3e443b65fa6ff502ec81da31bacdf44" exitCode=0 Oct 30 21:40:49.073329 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.073326 6720 generic.go:356] "Generic (PLEG): container finished" podID="1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b" containerID="4b2bd085af3f718bc596d63c5ad8ded90ba183b796157a3432e67072c75467f7" exitCode=0 Oct 30 21:40:49.073663 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.073330 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b","Type":"ContainerDied","Data":"a557f1280585b29505458f1f9fd878dceae79678a4f48ed21efc6d4c08bb82d8"} Oct 30 21:40:49.073663 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.073351 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:49.073663 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.073421 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b","Type":"ContainerDied","Data":"dd38eb45d35ddcf72610143f96e71fc7ee928e71fd3f8eeda09afc20bcc49e26"} Oct 30 21:40:49.073663 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.073436 6720 scope.go:122] "RemoveContainer" containerID="a557f1280585b29505458f1f9fd878dceae79678a4f48ed21efc6d4c08bb82d8" Oct 30 21:40:49.073663 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.073454 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b","Type":"ContainerDied","Data":"fd0dbf8a054c508e3dbab8f958d4fa7655b02969db228af9671e7c1cc9f9a883"} Oct 30 21:40:49.073663 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.073486 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b","Type":"ContainerDied","Data":"d894623a6390af62bd4043a10961f6e0c09665c6d5cf9e5694c540fca70213f4"} Oct 30 21:40:49.073663 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.073516 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b","Type":"ContainerDied","Data":"be222d32f0e90f6c8195bc3d07590a9be3e443b65fa6ff502ec81da31bacdf44"} Oct 30 21:40:49.073663 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.073542 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b","Type":"ContainerDied","Data":"4b2bd085af3f718bc596d63c5ad8ded90ba183b796157a3432e67072c75467f7"} Oct 30 21:40:49.073663 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.073581 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b","Type":"ContainerDied","Data":"7421daa4284e1388825935e1de921df314559c1141dd3cb2d1b6c2954e7d62c7"} Oct 30 21:40:49.083626 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.083608 6720 scope.go:122] "RemoveContainer" containerID="dd38eb45d35ddcf72610143f96e71fc7ee928e71fd3f8eeda09afc20bcc49e26" Oct 30 21:40:49.090985 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.090965 6720 scope.go:122] "RemoveContainer" containerID="fd0dbf8a054c508e3dbab8f958d4fa7655b02969db228af9671e7c1cc9f9a883" Oct 30 21:40:49.097525 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.097499 6720 scope.go:122] "RemoveContainer" containerID="d894623a6390af62bd4043a10961f6e0c09665c6d5cf9e5694c540fca70213f4" Oct 30 21:40:49.098525 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.098506 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 21:40:49.104364 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.104345 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 21:40:49.105620 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.105605 6720 scope.go:122] "RemoveContainer" containerID="be222d32f0e90f6c8195bc3d07590a9be3e443b65fa6ff502ec81da31bacdf44" Oct 30 21:40:49.112324 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.112310 6720 scope.go:122] "RemoveContainer" containerID="4b2bd085af3f718bc596d63c5ad8ded90ba183b796157a3432e67072c75467f7" Oct 30 21:40:49.119477 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.119460 6720 scope.go:122] "RemoveContainer" containerID="c9a2c4c7e354ff0bfe92d27ad47068f8f00d3163ec1f08a88a5b49ddd4cad80e" Oct 30 21:40:49.125947 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.125922 6720 scope.go:122] "RemoveContainer" containerID="a557f1280585b29505458f1f9fd878dceae79678a4f48ed21efc6d4c08bb82d8" Oct 30 21:40:49.126250 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:49.126222 6720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a557f1280585b29505458f1f9fd878dceae79678a4f48ed21efc6d4c08bb82d8\": container with ID starting with a557f1280585b29505458f1f9fd878dceae79678a4f48ed21efc6d4c08bb82d8 not found: ID does not exist" containerID="a557f1280585b29505458f1f9fd878dceae79678a4f48ed21efc6d4c08bb82d8" Oct 30 21:40:49.126315 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.126258 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a557f1280585b29505458f1f9fd878dceae79678a4f48ed21efc6d4c08bb82d8"} err="failed to get container status \"a557f1280585b29505458f1f9fd878dceae79678a4f48ed21efc6d4c08bb82d8\": rpc error: code = NotFound desc = could not find container \"a557f1280585b29505458f1f9fd878dceae79678a4f48ed21efc6d4c08bb82d8\": container with ID starting with a557f1280585b29505458f1f9fd878dceae79678a4f48ed21efc6d4c08bb82d8 not found: ID does not exist" Oct 30 21:40:49.126315 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.126289 6720 scope.go:122] "RemoveContainer" containerID="dd38eb45d35ddcf72610143f96e71fc7ee928e71fd3f8eeda09afc20bcc49e26" Oct 30 21:40:49.126641 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:49.126614 6720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd38eb45d35ddcf72610143f96e71fc7ee928e71fd3f8eeda09afc20bcc49e26\": container with ID starting with dd38eb45d35ddcf72610143f96e71fc7ee928e71fd3f8eeda09afc20bcc49e26 not found: ID does not exist" containerID="dd38eb45d35ddcf72610143f96e71fc7ee928e71fd3f8eeda09afc20bcc49e26" Oct 30 21:40:49.126737 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.126648 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd38eb45d35ddcf72610143f96e71fc7ee928e71fd3f8eeda09afc20bcc49e26"} err="failed to get container status \"dd38eb45d35ddcf72610143f96e71fc7ee928e71fd3f8eeda09afc20bcc49e26\": rpc error: code = NotFound desc = could not find container \"dd38eb45d35ddcf72610143f96e71fc7ee928e71fd3f8eeda09afc20bcc49e26\": container with ID starting with dd38eb45d35ddcf72610143f96e71fc7ee928e71fd3f8eeda09afc20bcc49e26 not found: ID does not exist" Oct 30 21:40:49.126737 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.126663 6720 scope.go:122] "RemoveContainer" containerID="fd0dbf8a054c508e3dbab8f958d4fa7655b02969db228af9671e7c1cc9f9a883" Oct 30 21:40:49.126969 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:49.126948 6720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd0dbf8a054c508e3dbab8f958d4fa7655b02969db228af9671e7c1cc9f9a883\": container with ID starting with fd0dbf8a054c508e3dbab8f958d4fa7655b02969db228af9671e7c1cc9f9a883 not found: ID does not exist" containerID="fd0dbf8a054c508e3dbab8f958d4fa7655b02969db228af9671e7c1cc9f9a883" Oct 30 21:40:49.127050 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.126972 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd0dbf8a054c508e3dbab8f958d4fa7655b02969db228af9671e7c1cc9f9a883"} err="failed to get container status \"fd0dbf8a054c508e3dbab8f958d4fa7655b02969db228af9671e7c1cc9f9a883\": rpc error: code = NotFound desc = could not find container \"fd0dbf8a054c508e3dbab8f958d4fa7655b02969db228af9671e7c1cc9f9a883\": container with ID starting with fd0dbf8a054c508e3dbab8f958d4fa7655b02969db228af9671e7c1cc9f9a883 not found: ID does not exist" Oct 30 21:40:49.127050 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.126986 6720 scope.go:122] "RemoveContainer" containerID="d894623a6390af62bd4043a10961f6e0c09665c6d5cf9e5694c540fca70213f4" Oct 30 21:40:49.127218 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:49.127203 6720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d894623a6390af62bd4043a10961f6e0c09665c6d5cf9e5694c540fca70213f4\": container with ID starting with d894623a6390af62bd4043a10961f6e0c09665c6d5cf9e5694c540fca70213f4 not found: ID does not exist" containerID="d894623a6390af62bd4043a10961f6e0c09665c6d5cf9e5694c540fca70213f4" Oct 30 21:40:49.127260 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.127222 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d894623a6390af62bd4043a10961f6e0c09665c6d5cf9e5694c540fca70213f4"} err="failed to get container status \"d894623a6390af62bd4043a10961f6e0c09665c6d5cf9e5694c540fca70213f4\": rpc error: code = NotFound desc = could not find container \"d894623a6390af62bd4043a10961f6e0c09665c6d5cf9e5694c540fca70213f4\": container with ID starting with d894623a6390af62bd4043a10961f6e0c09665c6d5cf9e5694c540fca70213f4 not found: ID does not exist" Oct 30 21:40:49.127260 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.127233 6720 scope.go:122] "RemoveContainer" containerID="be222d32f0e90f6c8195bc3d07590a9be3e443b65fa6ff502ec81da31bacdf44" Oct 30 21:40:49.127450 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:49.127435 6720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be222d32f0e90f6c8195bc3d07590a9be3e443b65fa6ff502ec81da31bacdf44\": container with ID starting with be222d32f0e90f6c8195bc3d07590a9be3e443b65fa6ff502ec81da31bacdf44 not found: ID does not exist" containerID="be222d32f0e90f6c8195bc3d07590a9be3e443b65fa6ff502ec81da31bacdf44" Oct 30 21:40:49.127514 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.127455 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be222d32f0e90f6c8195bc3d07590a9be3e443b65fa6ff502ec81da31bacdf44"} err="failed to get container status \"be222d32f0e90f6c8195bc3d07590a9be3e443b65fa6ff502ec81da31bacdf44\": rpc error: code = NotFound desc = could not find container \"be222d32f0e90f6c8195bc3d07590a9be3e443b65fa6ff502ec81da31bacdf44\": container with ID starting with be222d32f0e90f6c8195bc3d07590a9be3e443b65fa6ff502ec81da31bacdf44 not found: ID does not exist" Oct 30 21:40:49.127514 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.127468 6720 scope.go:122] "RemoveContainer" containerID="4b2bd085af3f718bc596d63c5ad8ded90ba183b796157a3432e67072c75467f7" Oct 30 21:40:49.127648 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:49.127633 6720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b2bd085af3f718bc596d63c5ad8ded90ba183b796157a3432e67072c75467f7\": container with ID starting with 4b2bd085af3f718bc596d63c5ad8ded90ba183b796157a3432e67072c75467f7 not found: ID does not exist" containerID="4b2bd085af3f718bc596d63c5ad8ded90ba183b796157a3432e67072c75467f7" Oct 30 21:40:49.127709 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.127650 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b2bd085af3f718bc596d63c5ad8ded90ba183b796157a3432e67072c75467f7"} err="failed to get container status \"4b2bd085af3f718bc596d63c5ad8ded90ba183b796157a3432e67072c75467f7\": rpc error: code = NotFound desc = could not find container \"4b2bd085af3f718bc596d63c5ad8ded90ba183b796157a3432e67072c75467f7\": container with ID starting with 4b2bd085af3f718bc596d63c5ad8ded90ba183b796157a3432e67072c75467f7 not found: ID does not exist" Oct 30 21:40:49.127709 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.127663 6720 scope.go:122] "RemoveContainer" containerID="c9a2c4c7e354ff0bfe92d27ad47068f8f00d3163ec1f08a88a5b49ddd4cad80e" Oct 30 21:40:49.127832 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:49.127819 6720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9a2c4c7e354ff0bfe92d27ad47068f8f00d3163ec1f08a88a5b49ddd4cad80e\": container with ID starting with c9a2c4c7e354ff0bfe92d27ad47068f8f00d3163ec1f08a88a5b49ddd4cad80e not found: ID does not exist" containerID="c9a2c4c7e354ff0bfe92d27ad47068f8f00d3163ec1f08a88a5b49ddd4cad80e" Oct 30 21:40:49.127869 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.127835 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9a2c4c7e354ff0bfe92d27ad47068f8f00d3163ec1f08a88a5b49ddd4cad80e"} err="failed to get container status \"c9a2c4c7e354ff0bfe92d27ad47068f8f00d3163ec1f08a88a5b49ddd4cad80e\": rpc error: code = NotFound desc = could not find container \"c9a2c4c7e354ff0bfe92d27ad47068f8f00d3163ec1f08a88a5b49ddd4cad80e\": container with ID starting with c9a2c4c7e354ff0bfe92d27ad47068f8f00d3163ec1f08a88a5b49ddd4cad80e not found: ID does not exist" Oct 30 21:40:49.127869 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.127845 6720 scope.go:122] "RemoveContainer" containerID="a557f1280585b29505458f1f9fd878dceae79678a4f48ed21efc6d4c08bb82d8" Oct 30 21:40:49.128421 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.128403 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a557f1280585b29505458f1f9fd878dceae79678a4f48ed21efc6d4c08bb82d8"} err="failed to get container status \"a557f1280585b29505458f1f9fd878dceae79678a4f48ed21efc6d4c08bb82d8\": rpc error: code = NotFound desc = could not find container \"a557f1280585b29505458f1f9fd878dceae79678a4f48ed21efc6d4c08bb82d8\": container with ID starting with a557f1280585b29505458f1f9fd878dceae79678a4f48ed21efc6d4c08bb82d8 not found: ID does not exist" Oct 30 21:40:49.128476 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.128421 6720 scope.go:122] "RemoveContainer" containerID="dd38eb45d35ddcf72610143f96e71fc7ee928e71fd3f8eeda09afc20bcc49e26" Oct 30 21:40:49.128605 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.128590 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd38eb45d35ddcf72610143f96e71fc7ee928e71fd3f8eeda09afc20bcc49e26"} err="failed to get container status \"dd38eb45d35ddcf72610143f96e71fc7ee928e71fd3f8eeda09afc20bcc49e26\": rpc error: code = NotFound desc = could not find container \"dd38eb45d35ddcf72610143f96e71fc7ee928e71fd3f8eeda09afc20bcc49e26\": container with ID starting with dd38eb45d35ddcf72610143f96e71fc7ee928e71fd3f8eeda09afc20bcc49e26 not found: ID does not exist" Oct 30 21:40:49.128644 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.128604 6720 scope.go:122] "RemoveContainer" containerID="fd0dbf8a054c508e3dbab8f958d4fa7655b02969db228af9671e7c1cc9f9a883" Oct 30 21:40:49.128817 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.128798 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd0dbf8a054c508e3dbab8f958d4fa7655b02969db228af9671e7c1cc9f9a883"} err="failed to get container status \"fd0dbf8a054c508e3dbab8f958d4fa7655b02969db228af9671e7c1cc9f9a883\": rpc error: code = NotFound desc = could not find container \"fd0dbf8a054c508e3dbab8f958d4fa7655b02969db228af9671e7c1cc9f9a883\": container with ID starting with fd0dbf8a054c508e3dbab8f958d4fa7655b02969db228af9671e7c1cc9f9a883 not found: ID does not exist" Oct 30 21:40:49.128859 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.128819 6720 scope.go:122] "RemoveContainer" containerID="d894623a6390af62bd4043a10961f6e0c09665c6d5cf9e5694c540fca70213f4" Oct 30 21:40:49.129027 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.129013 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d894623a6390af62bd4043a10961f6e0c09665c6d5cf9e5694c540fca70213f4"} err="failed to get container status \"d894623a6390af62bd4043a10961f6e0c09665c6d5cf9e5694c540fca70213f4\": rpc error: code = NotFound desc = could not find container \"d894623a6390af62bd4043a10961f6e0c09665c6d5cf9e5694c540fca70213f4\": container with ID starting with d894623a6390af62bd4043a10961f6e0c09665c6d5cf9e5694c540fca70213f4 not found: ID does not exist" Oct 30 21:40:49.129069 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.129027 6720 scope.go:122] "RemoveContainer" containerID="be222d32f0e90f6c8195bc3d07590a9be3e443b65fa6ff502ec81da31bacdf44" Oct 30 21:40:49.129240 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.129225 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be222d32f0e90f6c8195bc3d07590a9be3e443b65fa6ff502ec81da31bacdf44"} err="failed to get container status \"be222d32f0e90f6c8195bc3d07590a9be3e443b65fa6ff502ec81da31bacdf44\": rpc error: code = NotFound desc = could not find container \"be222d32f0e90f6c8195bc3d07590a9be3e443b65fa6ff502ec81da31bacdf44\": container with ID starting with be222d32f0e90f6c8195bc3d07590a9be3e443b65fa6ff502ec81da31bacdf44 not found: ID does not exist" Oct 30 21:40:49.129280 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.129239 6720 scope.go:122] "RemoveContainer" containerID="4b2bd085af3f718bc596d63c5ad8ded90ba183b796157a3432e67072c75467f7" Oct 30 21:40:49.129479 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.129461 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b2bd085af3f718bc596d63c5ad8ded90ba183b796157a3432e67072c75467f7"} err="failed to get container status \"4b2bd085af3f718bc596d63c5ad8ded90ba183b796157a3432e67072c75467f7\": rpc error: code = NotFound desc = could not find container \"4b2bd085af3f718bc596d63c5ad8ded90ba183b796157a3432e67072c75467f7\": container with ID starting with 4b2bd085af3f718bc596d63c5ad8ded90ba183b796157a3432e67072c75467f7 not found: ID does not exist" Oct 30 21:40:49.129530 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.129479 6720 scope.go:122] "RemoveContainer" containerID="c9a2c4c7e354ff0bfe92d27ad47068f8f00d3163ec1f08a88a5b49ddd4cad80e" Oct 30 21:40:49.129725 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.129708 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9a2c4c7e354ff0bfe92d27ad47068f8f00d3163ec1f08a88a5b49ddd4cad80e"} err="failed to get container status \"c9a2c4c7e354ff0bfe92d27ad47068f8f00d3163ec1f08a88a5b49ddd4cad80e\": rpc error: code = NotFound desc = could not find container \"c9a2c4c7e354ff0bfe92d27ad47068f8f00d3163ec1f08a88a5b49ddd4cad80e\": container with ID starting with c9a2c4c7e354ff0bfe92d27ad47068f8f00d3163ec1f08a88a5b49ddd4cad80e not found: ID does not exist" Oct 30 21:40:49.129725 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.129723 6720 scope.go:122] "RemoveContainer" containerID="a557f1280585b29505458f1f9fd878dceae79678a4f48ed21efc6d4c08bb82d8" Oct 30 21:40:49.129957 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.129936 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a557f1280585b29505458f1f9fd878dceae79678a4f48ed21efc6d4c08bb82d8"} err="failed to get container status \"a557f1280585b29505458f1f9fd878dceae79678a4f48ed21efc6d4c08bb82d8\": rpc error: code = NotFound desc = could not find container \"a557f1280585b29505458f1f9fd878dceae79678a4f48ed21efc6d4c08bb82d8\": container with ID starting with a557f1280585b29505458f1f9fd878dceae79678a4f48ed21efc6d4c08bb82d8 not found: ID does not exist" Oct 30 21:40:49.129997 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.129957 6720 scope.go:122] "RemoveContainer" containerID="dd38eb45d35ddcf72610143f96e71fc7ee928e71fd3f8eeda09afc20bcc49e26" Oct 30 21:40:49.130151 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.130132 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd38eb45d35ddcf72610143f96e71fc7ee928e71fd3f8eeda09afc20bcc49e26"} err="failed to get container status \"dd38eb45d35ddcf72610143f96e71fc7ee928e71fd3f8eeda09afc20bcc49e26\": rpc error: code = NotFound desc = could not find container \"dd38eb45d35ddcf72610143f96e71fc7ee928e71fd3f8eeda09afc20bcc49e26\": container with ID starting with dd38eb45d35ddcf72610143f96e71fc7ee928e71fd3f8eeda09afc20bcc49e26 not found: ID does not exist" Oct 30 21:40:49.130151 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.130150 6720 scope.go:122] "RemoveContainer" containerID="fd0dbf8a054c508e3dbab8f958d4fa7655b02969db228af9671e7c1cc9f9a883" Oct 30 21:40:49.130345 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.130330 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd0dbf8a054c508e3dbab8f958d4fa7655b02969db228af9671e7c1cc9f9a883"} err="failed to get container status \"fd0dbf8a054c508e3dbab8f958d4fa7655b02969db228af9671e7c1cc9f9a883\": rpc error: code = NotFound desc = could not find container \"fd0dbf8a054c508e3dbab8f958d4fa7655b02969db228af9671e7c1cc9f9a883\": container with ID starting with fd0dbf8a054c508e3dbab8f958d4fa7655b02969db228af9671e7c1cc9f9a883 not found: ID does not exist" Oct 30 21:40:49.130410 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.130345 6720 scope.go:122] "RemoveContainer" containerID="d894623a6390af62bd4043a10961f6e0c09665c6d5cf9e5694c540fca70213f4" Oct 30 21:40:49.130562 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.130548 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d894623a6390af62bd4043a10961f6e0c09665c6d5cf9e5694c540fca70213f4"} err="failed to get container status \"d894623a6390af62bd4043a10961f6e0c09665c6d5cf9e5694c540fca70213f4\": rpc error: code = NotFound desc = could not find container \"d894623a6390af62bd4043a10961f6e0c09665c6d5cf9e5694c540fca70213f4\": container with ID starting with d894623a6390af62bd4043a10961f6e0c09665c6d5cf9e5694c540fca70213f4 not found: ID does not exist" Oct 30 21:40:49.130608 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.130562 6720 scope.go:122] "RemoveContainer" containerID="be222d32f0e90f6c8195bc3d07590a9be3e443b65fa6ff502ec81da31bacdf44" Oct 30 21:40:49.130740 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.130726 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be222d32f0e90f6c8195bc3d07590a9be3e443b65fa6ff502ec81da31bacdf44"} err="failed to get container status \"be222d32f0e90f6c8195bc3d07590a9be3e443b65fa6ff502ec81da31bacdf44\": rpc error: code = NotFound desc = could not find container \"be222d32f0e90f6c8195bc3d07590a9be3e443b65fa6ff502ec81da31bacdf44\": container with ID starting with be222d32f0e90f6c8195bc3d07590a9be3e443b65fa6ff502ec81da31bacdf44 not found: ID does not exist" Oct 30 21:40:49.130796 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.130742 6720 scope.go:122] "RemoveContainer" containerID="4b2bd085af3f718bc596d63c5ad8ded90ba183b796157a3432e67072c75467f7" Oct 30 21:40:49.130977 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.130964 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b2bd085af3f718bc596d63c5ad8ded90ba183b796157a3432e67072c75467f7"} err="failed to get container status \"4b2bd085af3f718bc596d63c5ad8ded90ba183b796157a3432e67072c75467f7\": rpc error: code = NotFound desc = could not find container \"4b2bd085af3f718bc596d63c5ad8ded90ba183b796157a3432e67072c75467f7\": container with ID starting with 4b2bd085af3f718bc596d63c5ad8ded90ba183b796157a3432e67072c75467f7 not found: ID does not exist" Oct 30 21:40:49.130977 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.130977 6720 scope.go:122] "RemoveContainer" containerID="c9a2c4c7e354ff0bfe92d27ad47068f8f00d3163ec1f08a88a5b49ddd4cad80e" Oct 30 21:40:49.131163 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.131149 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9a2c4c7e354ff0bfe92d27ad47068f8f00d3163ec1f08a88a5b49ddd4cad80e"} err="failed to get container status \"c9a2c4c7e354ff0bfe92d27ad47068f8f00d3163ec1f08a88a5b49ddd4cad80e\": rpc error: code = NotFound desc = could not find container \"c9a2c4c7e354ff0bfe92d27ad47068f8f00d3163ec1f08a88a5b49ddd4cad80e\": container with ID starting with c9a2c4c7e354ff0bfe92d27ad47068f8f00d3163ec1f08a88a5b49ddd4cad80e not found: ID does not exist" Oct 30 21:40:49.131210 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.131163 6720 scope.go:122] "RemoveContainer" containerID="a557f1280585b29505458f1f9fd878dceae79678a4f48ed21efc6d4c08bb82d8" Oct 30 21:40:49.131340 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.131326 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a557f1280585b29505458f1f9fd878dceae79678a4f48ed21efc6d4c08bb82d8"} err="failed to get container status \"a557f1280585b29505458f1f9fd878dceae79678a4f48ed21efc6d4c08bb82d8\": rpc error: code = NotFound desc = could not find container \"a557f1280585b29505458f1f9fd878dceae79678a4f48ed21efc6d4c08bb82d8\": container with ID starting with a557f1280585b29505458f1f9fd878dceae79678a4f48ed21efc6d4c08bb82d8 not found: ID does not exist" Oct 30 21:40:49.131402 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.131340 6720 scope.go:122] "RemoveContainer" containerID="dd38eb45d35ddcf72610143f96e71fc7ee928e71fd3f8eeda09afc20bcc49e26" Oct 30 21:40:49.131545 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.131525 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd38eb45d35ddcf72610143f96e71fc7ee928e71fd3f8eeda09afc20bcc49e26"} err="failed to get container status \"dd38eb45d35ddcf72610143f96e71fc7ee928e71fd3f8eeda09afc20bcc49e26\": rpc error: code = NotFound desc = could not find container \"dd38eb45d35ddcf72610143f96e71fc7ee928e71fd3f8eeda09afc20bcc49e26\": container with ID starting with dd38eb45d35ddcf72610143f96e71fc7ee928e71fd3f8eeda09afc20bcc49e26 not found: ID does not exist" Oct 30 21:40:49.131615 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.131546 6720 scope.go:122] "RemoveContainer" containerID="fd0dbf8a054c508e3dbab8f958d4fa7655b02969db228af9671e7c1cc9f9a883" Oct 30 21:40:49.131771 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.131755 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd0dbf8a054c508e3dbab8f958d4fa7655b02969db228af9671e7c1cc9f9a883"} err="failed to get container status \"fd0dbf8a054c508e3dbab8f958d4fa7655b02969db228af9671e7c1cc9f9a883\": rpc error: code = NotFound desc = could not find container \"fd0dbf8a054c508e3dbab8f958d4fa7655b02969db228af9671e7c1cc9f9a883\": container with ID starting with fd0dbf8a054c508e3dbab8f958d4fa7655b02969db228af9671e7c1cc9f9a883 not found: ID does not exist" Oct 30 21:40:49.131816 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.131771 6720 scope.go:122] "RemoveContainer" containerID="d894623a6390af62bd4043a10961f6e0c09665c6d5cf9e5694c540fca70213f4" Oct 30 21:40:49.131954 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.131935 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d894623a6390af62bd4043a10961f6e0c09665c6d5cf9e5694c540fca70213f4"} err="failed to get container status \"d894623a6390af62bd4043a10961f6e0c09665c6d5cf9e5694c540fca70213f4\": rpc error: code = NotFound desc = could not find container \"d894623a6390af62bd4043a10961f6e0c09665c6d5cf9e5694c540fca70213f4\": container with ID starting with d894623a6390af62bd4043a10961f6e0c09665c6d5cf9e5694c540fca70213f4 not found: ID does not exist" Oct 30 21:40:49.132016 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.131956 6720 scope.go:122] "RemoveContainer" containerID="be222d32f0e90f6c8195bc3d07590a9be3e443b65fa6ff502ec81da31bacdf44" Oct 30 21:40:49.132212 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.132193 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be222d32f0e90f6c8195bc3d07590a9be3e443b65fa6ff502ec81da31bacdf44"} err="failed to get container status \"be222d32f0e90f6c8195bc3d07590a9be3e443b65fa6ff502ec81da31bacdf44\": rpc error: code = NotFound desc = could not find container \"be222d32f0e90f6c8195bc3d07590a9be3e443b65fa6ff502ec81da31bacdf44\": container with ID starting with be222d32f0e90f6c8195bc3d07590a9be3e443b65fa6ff502ec81da31bacdf44 not found: ID does not exist" Oct 30 21:40:49.132212 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.132210 6720 scope.go:122] "RemoveContainer" containerID="4b2bd085af3f718bc596d63c5ad8ded90ba183b796157a3432e67072c75467f7" Oct 30 21:40:49.132428 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.132413 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b2bd085af3f718bc596d63c5ad8ded90ba183b796157a3432e67072c75467f7"} err="failed to get container status \"4b2bd085af3f718bc596d63c5ad8ded90ba183b796157a3432e67072c75467f7\": rpc error: code = NotFound desc = could not find container \"4b2bd085af3f718bc596d63c5ad8ded90ba183b796157a3432e67072c75467f7\": container with ID starting with 4b2bd085af3f718bc596d63c5ad8ded90ba183b796157a3432e67072c75467f7 not found: ID does not exist" Oct 30 21:40:49.132476 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.132429 6720 scope.go:122] "RemoveContainer" containerID="c9a2c4c7e354ff0bfe92d27ad47068f8f00d3163ec1f08a88a5b49ddd4cad80e" Oct 30 21:40:49.132629 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.132613 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9a2c4c7e354ff0bfe92d27ad47068f8f00d3163ec1f08a88a5b49ddd4cad80e"} err="failed to get container status \"c9a2c4c7e354ff0bfe92d27ad47068f8f00d3163ec1f08a88a5b49ddd4cad80e\": rpc error: code = NotFound desc = could not find container \"c9a2c4c7e354ff0bfe92d27ad47068f8f00d3163ec1f08a88a5b49ddd4cad80e\": container with ID starting with c9a2c4c7e354ff0bfe92d27ad47068f8f00d3163ec1f08a88a5b49ddd4cad80e not found: ID does not exist" Oct 30 21:40:49.132670 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.132629 6720 scope.go:122] "RemoveContainer" containerID="a557f1280585b29505458f1f9fd878dceae79678a4f48ed21efc6d4c08bb82d8" Oct 30 21:40:49.132790 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.132776 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a557f1280585b29505458f1f9fd878dceae79678a4f48ed21efc6d4c08bb82d8"} err="failed to get container status \"a557f1280585b29505458f1f9fd878dceae79678a4f48ed21efc6d4c08bb82d8\": rpc error: code = NotFound desc = could not find container \"a557f1280585b29505458f1f9fd878dceae79678a4f48ed21efc6d4c08bb82d8\": container with ID starting with a557f1280585b29505458f1f9fd878dceae79678a4f48ed21efc6d4c08bb82d8 not found: ID does not exist" Oct 30 21:40:49.132825 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.132791 6720 scope.go:122] "RemoveContainer" containerID="dd38eb45d35ddcf72610143f96e71fc7ee928e71fd3f8eeda09afc20bcc49e26" Oct 30 21:40:49.133035 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.133016 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd38eb45d35ddcf72610143f96e71fc7ee928e71fd3f8eeda09afc20bcc49e26"} err="failed to get container status \"dd38eb45d35ddcf72610143f96e71fc7ee928e71fd3f8eeda09afc20bcc49e26\": rpc error: code = NotFound desc = could not find container \"dd38eb45d35ddcf72610143f96e71fc7ee928e71fd3f8eeda09afc20bcc49e26\": container with ID starting with dd38eb45d35ddcf72610143f96e71fc7ee928e71fd3f8eeda09afc20bcc49e26 not found: ID does not exist" Oct 30 21:40:49.133035 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.133033 6720 scope.go:122] "RemoveContainer" containerID="fd0dbf8a054c508e3dbab8f958d4fa7655b02969db228af9671e7c1cc9f9a883" Oct 30 21:40:49.133305 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.133282 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd0dbf8a054c508e3dbab8f958d4fa7655b02969db228af9671e7c1cc9f9a883"} err="failed to get container status \"fd0dbf8a054c508e3dbab8f958d4fa7655b02969db228af9671e7c1cc9f9a883\": rpc error: code = NotFound desc = could not find container \"fd0dbf8a054c508e3dbab8f958d4fa7655b02969db228af9671e7c1cc9f9a883\": container with ID starting with fd0dbf8a054c508e3dbab8f958d4fa7655b02969db228af9671e7c1cc9f9a883 not found: ID does not exist" Oct 30 21:40:49.133401 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.133308 6720 scope.go:122] "RemoveContainer" containerID="d894623a6390af62bd4043a10961f6e0c09665c6d5cf9e5694c540fca70213f4" Oct 30 21:40:49.133701 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.133667 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d894623a6390af62bd4043a10961f6e0c09665c6d5cf9e5694c540fca70213f4"} err="failed to get container status \"d894623a6390af62bd4043a10961f6e0c09665c6d5cf9e5694c540fca70213f4\": rpc error: code = NotFound desc = could not find container \"d894623a6390af62bd4043a10961f6e0c09665c6d5cf9e5694c540fca70213f4\": container with ID starting with d894623a6390af62bd4043a10961f6e0c09665c6d5cf9e5694c540fca70213f4 not found: ID does not exist" Oct 30 21:40:49.133701 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.133693 6720 scope.go:122] "RemoveContainer" containerID="be222d32f0e90f6c8195bc3d07590a9be3e443b65fa6ff502ec81da31bacdf44" Oct 30 21:40:49.135562 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.135530 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be222d32f0e90f6c8195bc3d07590a9be3e443b65fa6ff502ec81da31bacdf44"} err="failed to get container status \"be222d32f0e90f6c8195bc3d07590a9be3e443b65fa6ff502ec81da31bacdf44\": rpc error: code = NotFound desc = could not find container \"be222d32f0e90f6c8195bc3d07590a9be3e443b65fa6ff502ec81da31bacdf44\": container with ID starting with be222d32f0e90f6c8195bc3d07590a9be3e443b65fa6ff502ec81da31bacdf44 not found: ID does not exist" Oct 30 21:40:49.135671 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.135562 6720 scope.go:122] "RemoveContainer" containerID="4b2bd085af3f718bc596d63c5ad8ded90ba183b796157a3432e67072c75467f7" Oct 30 21:40:49.135971 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.135942 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b2bd085af3f718bc596d63c5ad8ded90ba183b796157a3432e67072c75467f7"} err="failed to get container status \"4b2bd085af3f718bc596d63c5ad8ded90ba183b796157a3432e67072c75467f7\": rpc error: code = NotFound desc = could not find container \"4b2bd085af3f718bc596d63c5ad8ded90ba183b796157a3432e67072c75467f7\": container with ID starting with 4b2bd085af3f718bc596d63c5ad8ded90ba183b796157a3432e67072c75467f7 not found: ID does not exist" Oct 30 21:40:49.136179 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.135972 6720 scope.go:122] "RemoveContainer" containerID="c9a2c4c7e354ff0bfe92d27ad47068f8f00d3163ec1f08a88a5b49ddd4cad80e" Oct 30 21:40:49.136420 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.136379 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 21:40:49.136491 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.136457 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9a2c4c7e354ff0bfe92d27ad47068f8f00d3163ec1f08a88a5b49ddd4cad80e"} err="failed to get container status \"c9a2c4c7e354ff0bfe92d27ad47068f8f00d3163ec1f08a88a5b49ddd4cad80e\": rpc error: code = NotFound desc = could not find container \"c9a2c4c7e354ff0bfe92d27ad47068f8f00d3163ec1f08a88a5b49ddd4cad80e\": container with ID starting with c9a2c4c7e354ff0bfe92d27ad47068f8f00d3163ec1f08a88a5b49ddd4cad80e not found: ID does not exist" Oct 30 21:40:49.136491 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.136479 6720 scope.go:122] "RemoveContainer" containerID="a557f1280585b29505458f1f9fd878dceae79678a4f48ed21efc6d4c08bb82d8" Oct 30 21:40:49.136838 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.136809 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a557f1280585b29505458f1f9fd878dceae79678a4f48ed21efc6d4c08bb82d8"} err="failed to get container status \"a557f1280585b29505458f1f9fd878dceae79678a4f48ed21efc6d4c08bb82d8\": rpc error: code = NotFound desc = could not find container \"a557f1280585b29505458f1f9fd878dceae79678a4f48ed21efc6d4c08bb82d8\": container with ID starting with a557f1280585b29505458f1f9fd878dceae79678a4f48ed21efc6d4c08bb82d8 not found: ID does not exist" Oct 30 21:40:49.136901 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.136840 6720 scope.go:122] "RemoveContainer" containerID="dd38eb45d35ddcf72610143f96e71fc7ee928e71fd3f8eeda09afc20bcc49e26" Oct 30 21:40:49.136995 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.136978 6720 cpu_manager.go:414] "containerMap: removing container" podUID="1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b" containerName="init-config-reloader" Oct 30 21:40:49.137040 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.136998 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b" containerName="init-config-reloader" Oct 30 21:40:49.137040 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.137020 6720 cpu_manager.go:414] "containerMap: removing container" podUID="1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b" containerName="kube-rbac-proxy" Oct 30 21:40:49.137040 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.137027 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b" containerName="kube-rbac-proxy" Oct 30 21:40:49.137201 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.137042 6720 cpu_manager.go:414] "containerMap: removing container" podUID="1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b" containerName="config-reloader" Oct 30 21:40:49.137201 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.137051 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b" containerName="config-reloader" Oct 30 21:40:49.137201 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.137072 6720 cpu_manager.go:414] "containerMap: removing container" podUID="1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b" containerName="kube-rbac-proxy-web" Oct 30 21:40:49.137201 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.137079 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b" containerName="kube-rbac-proxy-web" Oct 30 21:40:49.137201 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.137093 6720 cpu_manager.go:414] "containerMap: removing container" podUID="1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b" containerName="prometheus" Oct 30 21:40:49.137201 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.137100 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b" containerName="prometheus" Oct 30 21:40:49.137201 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.137127 6720 cpu_manager.go:414] "containerMap: removing container" podUID="1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b" containerName="kube-rbac-proxy-thanos" Oct 30 21:40:49.137201 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.137134 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b" containerName="kube-rbac-proxy-thanos" Oct 30 21:40:49.137201 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.137150 6720 cpu_manager.go:414] "containerMap: removing container" podUID="1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b" containerName="thanos-sidecar" Oct 30 21:40:49.137201 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.137158 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b" containerName="thanos-sidecar" Oct 30 21:40:49.137663 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.137272 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b" containerName="kube-rbac-proxy-thanos" Oct 30 21:40:49.137663 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.137290 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b" containerName="thanos-sidecar" Oct 30 21:40:49.137663 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.137302 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b" containerName="init-config-reloader" Oct 30 21:40:49.137663 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.137312 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b" containerName="kube-rbac-proxy" Oct 30 21:40:49.137663 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.137325 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b" containerName="config-reloader" Oct 30 21:40:49.137663 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.137335 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b" containerName="kube-rbac-proxy-web" Oct 30 21:40:49.137663 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.137342 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b" containerName="prometheus" Oct 30 21:40:49.139732 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.139698 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd38eb45d35ddcf72610143f96e71fc7ee928e71fd3f8eeda09afc20bcc49e26"} err="failed to get container status \"dd38eb45d35ddcf72610143f96e71fc7ee928e71fd3f8eeda09afc20bcc49e26\": rpc error: code = NotFound desc = could not find container \"dd38eb45d35ddcf72610143f96e71fc7ee928e71fd3f8eeda09afc20bcc49e26\": container with ID starting with dd38eb45d35ddcf72610143f96e71fc7ee928e71fd3f8eeda09afc20bcc49e26 not found: ID does not exist" Oct 30 21:40:49.139732 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.139731 6720 scope.go:122] "RemoveContainer" containerID="fd0dbf8a054c508e3dbab8f958d4fa7655b02969db228af9671e7c1cc9f9a883" Oct 30 21:40:49.140317 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.140288 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd0dbf8a054c508e3dbab8f958d4fa7655b02969db228af9671e7c1cc9f9a883"} err="failed to get container status \"fd0dbf8a054c508e3dbab8f958d4fa7655b02969db228af9671e7c1cc9f9a883\": rpc error: code = NotFound desc = could not find container \"fd0dbf8a054c508e3dbab8f958d4fa7655b02969db228af9671e7c1cc9f9a883\": container with ID starting with fd0dbf8a054c508e3dbab8f958d4fa7655b02969db228af9671e7c1cc9f9a883 not found: ID does not exist" Oct 30 21:40:49.140422 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.140318 6720 scope.go:122] "RemoveContainer" containerID="d894623a6390af62bd4043a10961f6e0c09665c6d5cf9e5694c540fca70213f4" Oct 30 21:40:49.141005 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.140973 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d894623a6390af62bd4043a10961f6e0c09665c6d5cf9e5694c540fca70213f4"} err="failed to get container status \"d894623a6390af62bd4043a10961f6e0c09665c6d5cf9e5694c540fca70213f4\": rpc error: code = NotFound desc = could not find container \"d894623a6390af62bd4043a10961f6e0c09665c6d5cf9e5694c540fca70213f4\": container with ID starting with d894623a6390af62bd4043a10961f6e0c09665c6d5cf9e5694c540fca70213f4 not found: ID does not exist" Oct 30 21:40:49.141005 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.141005 6720 scope.go:122] "RemoveContainer" containerID="be222d32f0e90f6c8195bc3d07590a9be3e443b65fa6ff502ec81da31bacdf44" Oct 30 21:40:49.141300 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.141272 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be222d32f0e90f6c8195bc3d07590a9be3e443b65fa6ff502ec81da31bacdf44"} err="failed to get container status \"be222d32f0e90f6c8195bc3d07590a9be3e443b65fa6ff502ec81da31bacdf44\": rpc error: code = NotFound desc = could not find container \"be222d32f0e90f6c8195bc3d07590a9be3e443b65fa6ff502ec81da31bacdf44\": container with ID starting with be222d32f0e90f6c8195bc3d07590a9be3e443b65fa6ff502ec81da31bacdf44 not found: ID does not exist" Oct 30 21:40:49.141300 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.141292 6720 scope.go:122] "RemoveContainer" containerID="4b2bd085af3f718bc596d63c5ad8ded90ba183b796157a3432e67072c75467f7" Oct 30 21:40:49.141561 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.141544 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b2bd085af3f718bc596d63c5ad8ded90ba183b796157a3432e67072c75467f7"} err="failed to get container status \"4b2bd085af3f718bc596d63c5ad8ded90ba183b796157a3432e67072c75467f7\": rpc error: code = NotFound desc = could not find container \"4b2bd085af3f718bc596d63c5ad8ded90ba183b796157a3432e67072c75467f7\": container with ID starting with 4b2bd085af3f718bc596d63c5ad8ded90ba183b796157a3432e67072c75467f7 not found: ID does not exist" Oct 30 21:40:49.141561 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.141562 6720 scope.go:122] "RemoveContainer" containerID="c9a2c4c7e354ff0bfe92d27ad47068f8f00d3163ec1f08a88a5b49ddd4cad80e" Oct 30 21:40:49.141845 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.141816 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9a2c4c7e354ff0bfe92d27ad47068f8f00d3163ec1f08a88a5b49ddd4cad80e"} err="failed to get container status \"c9a2c4c7e354ff0bfe92d27ad47068f8f00d3163ec1f08a88a5b49ddd4cad80e\": rpc error: code = NotFound desc = could not find container \"c9a2c4c7e354ff0bfe92d27ad47068f8f00d3163ec1f08a88a5b49ddd4cad80e\": container with ID starting with c9a2c4c7e354ff0bfe92d27ad47068f8f00d3163ec1f08a88a5b49ddd4cad80e not found: ID does not exist" Oct 30 21:40:49.145502 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.145480 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:49.150271 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.150249 6720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 21:40:49.150354 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.150274 6720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 21:40:49.152039 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.152018 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 21:40:49.152166 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.152060 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 21:40:49.152166 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.152080 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 21:40:49.152166 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.152095 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 21:40:49.152166 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.152127 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 21:40:49.152166 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.152138 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 21:40:49.152166 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.152129 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 21:40:49.153657 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.153616 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 21:40:49.153924 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.153661 6720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 21:40:49.153924 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.153732 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 21:40:49.153924 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.153782 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-6slhptju3coep\"" Oct 30 21:40:49.155617 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.155601 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-f6sd2\"" Oct 30 21:40:49.159306 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.159286 6720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 21:40:49.159424 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.159406 6720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 21:40:49.161624 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.161605 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 21:40:49.288309 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.288273 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/163b57d1-c2a7-4939-8f82-77c01cd1721b-web-config\") pod \"prometheus-k8s-0\" (UID: \"163b57d1-c2a7-4939-8f82-77c01cd1721b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:49.288309 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.288302 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/163b57d1-c2a7-4939-8f82-77c01cd1721b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"163b57d1-c2a7-4939-8f82-77c01cd1721b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:49.288828 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.288431 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/163b57d1-c2a7-4939-8f82-77c01cd1721b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"163b57d1-c2a7-4939-8f82-77c01cd1721b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:49.288828 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.288473 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/163b57d1-c2a7-4939-8f82-77c01cd1721b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"163b57d1-c2a7-4939-8f82-77c01cd1721b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:49.288828 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.288513 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/163b57d1-c2a7-4939-8f82-77c01cd1721b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"163b57d1-c2a7-4939-8f82-77c01cd1721b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:49.288828 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.288529 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/163b57d1-c2a7-4939-8f82-77c01cd1721b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"163b57d1-c2a7-4939-8f82-77c01cd1721b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:49.288828 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.288616 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/163b57d1-c2a7-4939-8f82-77c01cd1721b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"163b57d1-c2a7-4939-8f82-77c01cd1721b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:49.288828 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.288641 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/163b57d1-c2a7-4939-8f82-77c01cd1721b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"163b57d1-c2a7-4939-8f82-77c01cd1721b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:49.288828 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.288657 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/163b57d1-c2a7-4939-8f82-77c01cd1721b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"163b57d1-c2a7-4939-8f82-77c01cd1721b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:49.288828 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.288703 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/163b57d1-c2a7-4939-8f82-77c01cd1721b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"163b57d1-c2a7-4939-8f82-77c01cd1721b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:49.288828 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.288743 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/163b57d1-c2a7-4939-8f82-77c01cd1721b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"163b57d1-c2a7-4939-8f82-77c01cd1721b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:49.289273 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.288850 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/163b57d1-c2a7-4939-8f82-77c01cd1721b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"163b57d1-c2a7-4939-8f82-77c01cd1721b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:49.289273 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.288874 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/163b57d1-c2a7-4939-8f82-77c01cd1721b-config\") pod \"prometheus-k8s-0\" (UID: \"163b57d1-c2a7-4939-8f82-77c01cd1721b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:49.289273 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.288903 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/163b57d1-c2a7-4939-8f82-77c01cd1721b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"163b57d1-c2a7-4939-8f82-77c01cd1721b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:49.289273 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.288929 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/163b57d1-c2a7-4939-8f82-77c01cd1721b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"163b57d1-c2a7-4939-8f82-77c01cd1721b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:49.289273 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.288952 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz24r\" (UniqueName: \"kubernetes.io/projected/163b57d1-c2a7-4939-8f82-77c01cd1721b-kube-api-access-kz24r\") pod \"prometheus-k8s-0\" (UID: \"163b57d1-c2a7-4939-8f82-77c01cd1721b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:49.289273 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.289008 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/163b57d1-c2a7-4939-8f82-77c01cd1721b-config-out\") pod \"prometheus-k8s-0\" (UID: \"163b57d1-c2a7-4939-8f82-77c01cd1721b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:49.289273 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.289049 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/163b57d1-c2a7-4939-8f82-77c01cd1721b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"163b57d1-c2a7-4939-8f82-77c01cd1721b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:49.289273 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.289092 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/163b57d1-c2a7-4939-8f82-77c01cd1721b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"163b57d1-c2a7-4939-8f82-77c01cd1721b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:49.289273 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.289112 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/163b57d1-c2a7-4939-8f82-77c01cd1721b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"163b57d1-c2a7-4939-8f82-77c01cd1721b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:49.389858 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.389828 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/163b57d1-c2a7-4939-8f82-77c01cd1721b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"163b57d1-c2a7-4939-8f82-77c01cd1721b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:49.389995 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.389861 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/163b57d1-c2a7-4939-8f82-77c01cd1721b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"163b57d1-c2a7-4939-8f82-77c01cd1721b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:49.389995 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.389884 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/163b57d1-c2a7-4939-8f82-77c01cd1721b-web-config\") pod \"prometheus-k8s-0\" (UID: \"163b57d1-c2a7-4939-8f82-77c01cd1721b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:49.389995 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.389905 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/163b57d1-c2a7-4939-8f82-77c01cd1721b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"163b57d1-c2a7-4939-8f82-77c01cd1721b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:49.390112 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.390041 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/163b57d1-c2a7-4939-8f82-77c01cd1721b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"163b57d1-c2a7-4939-8f82-77c01cd1721b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:49.390112 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.390093 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/163b57d1-c2a7-4939-8f82-77c01cd1721b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"163b57d1-c2a7-4939-8f82-77c01cd1721b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:49.390183 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.390139 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/163b57d1-c2a7-4939-8f82-77c01cd1721b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"163b57d1-c2a7-4939-8f82-77c01cd1721b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:49.390183 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.390159 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/163b57d1-c2a7-4939-8f82-77c01cd1721b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"163b57d1-c2a7-4939-8f82-77c01cd1721b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:49.390283 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.390194 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/163b57d1-c2a7-4939-8f82-77c01cd1721b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"163b57d1-c2a7-4939-8f82-77c01cd1721b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:49.390283 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.390217 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/163b57d1-c2a7-4939-8f82-77c01cd1721b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"163b57d1-c2a7-4939-8f82-77c01cd1721b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:49.390283 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.390219 6720 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 21:40:49.390283 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.390241 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/163b57d1-c2a7-4939-8f82-77c01cd1721b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"163b57d1-c2a7-4939-8f82-77c01cd1721b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:49.390283 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.390266 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/163b57d1-c2a7-4939-8f82-77c01cd1721b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"163b57d1-c2a7-4939-8f82-77c01cd1721b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:49.390570 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.390305 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/163b57d1-c2a7-4939-8f82-77c01cd1721b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"163b57d1-c2a7-4939-8f82-77c01cd1721b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:49.390570 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.390348 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/163b57d1-c2a7-4939-8f82-77c01cd1721b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"163b57d1-c2a7-4939-8f82-77c01cd1721b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:49.390570 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.390418 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/163b57d1-c2a7-4939-8f82-77c01cd1721b-config\") pod \"prometheus-k8s-0\" (UID: \"163b57d1-c2a7-4939-8f82-77c01cd1721b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:49.390570 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.390459 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/163b57d1-c2a7-4939-8f82-77c01cd1721b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"163b57d1-c2a7-4939-8f82-77c01cd1721b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:49.390570 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.390491 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/163b57d1-c2a7-4939-8f82-77c01cd1721b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"163b57d1-c2a7-4939-8f82-77c01cd1721b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:49.390570 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.390509 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/163b57d1-c2a7-4939-8f82-77c01cd1721b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"163b57d1-c2a7-4939-8f82-77c01cd1721b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:49.391070 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.390744 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/163b57d1-c2a7-4939-8f82-77c01cd1721b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"163b57d1-c2a7-4939-8f82-77c01cd1721b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:49.391070 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.390744 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/163b57d1-c2a7-4939-8f82-77c01cd1721b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"163b57d1-c2a7-4939-8f82-77c01cd1721b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:49.391070 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.390514 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz24r\" (UniqueName: \"kubernetes.io/projected/163b57d1-c2a7-4939-8f82-77c01cd1721b-kube-api-access-kz24r\") pod \"prometheus-k8s-0\" (UID: \"163b57d1-c2a7-4939-8f82-77c01cd1721b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:49.391070 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.390884 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/163b57d1-c2a7-4939-8f82-77c01cd1721b-config-out\") pod \"prometheus-k8s-0\" (UID: \"163b57d1-c2a7-4939-8f82-77c01cd1721b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:49.391070 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.390956 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/163b57d1-c2a7-4939-8f82-77c01cd1721b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"163b57d1-c2a7-4939-8f82-77c01cd1721b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:49.391525 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.391495 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/163b57d1-c2a7-4939-8f82-77c01cd1721b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"163b57d1-c2a7-4939-8f82-77c01cd1721b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:49.393521 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.393496 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/163b57d1-c2a7-4939-8f82-77c01cd1721b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"163b57d1-c2a7-4939-8f82-77c01cd1721b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:49.393630 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.393587 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/163b57d1-c2a7-4939-8f82-77c01cd1721b-config-out\") pod \"prometheus-k8s-0\" (UID: \"163b57d1-c2a7-4939-8f82-77c01cd1721b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:49.393777 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.393754 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/163b57d1-c2a7-4939-8f82-77c01cd1721b-config\") pod \"prometheus-k8s-0\" (UID: \"163b57d1-c2a7-4939-8f82-77c01cd1721b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:49.393865 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.393844 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/163b57d1-c2a7-4939-8f82-77c01cd1721b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"163b57d1-c2a7-4939-8f82-77c01cd1721b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:49.394182 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.393873 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/163b57d1-c2a7-4939-8f82-77c01cd1721b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"163b57d1-c2a7-4939-8f82-77c01cd1721b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:49.394429 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.394378 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/163b57d1-c2a7-4939-8f82-77c01cd1721b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"163b57d1-c2a7-4939-8f82-77c01cd1721b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:49.394863 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.394699 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/163b57d1-c2a7-4939-8f82-77c01cd1721b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"163b57d1-c2a7-4939-8f82-77c01cd1721b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:49.395102 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.395078 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/163b57d1-c2a7-4939-8f82-77c01cd1721b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"163b57d1-c2a7-4939-8f82-77c01cd1721b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:49.395237 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.395214 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/163b57d1-c2a7-4939-8f82-77c01cd1721b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"163b57d1-c2a7-4939-8f82-77c01cd1721b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:49.395781 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.395750 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/163b57d1-c2a7-4939-8f82-77c01cd1721b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"163b57d1-c2a7-4939-8f82-77c01cd1721b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:49.395907 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.395884 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/163b57d1-c2a7-4939-8f82-77c01cd1721b-web-config\") pod \"prometheus-k8s-0\" (UID: \"163b57d1-c2a7-4939-8f82-77c01cd1721b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:49.396011 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.395991 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/163b57d1-c2a7-4939-8f82-77c01cd1721b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"163b57d1-c2a7-4939-8f82-77c01cd1721b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:49.396226 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.396206 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/163b57d1-c2a7-4939-8f82-77c01cd1721b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"163b57d1-c2a7-4939-8f82-77c01cd1721b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:49.396368 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.396349 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/163b57d1-c2a7-4939-8f82-77c01cd1721b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"163b57d1-c2a7-4939-8f82-77c01cd1721b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:49.396883 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.396862 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/163b57d1-c2a7-4939-8f82-77c01cd1721b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"163b57d1-c2a7-4939-8f82-77c01cd1721b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:49.403296 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.403275 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz24r\" (UniqueName: \"kubernetes.io/projected/163b57d1-c2a7-4939-8f82-77c01cd1721b-kube-api-access-kz24r\") pod \"prometheus-k8s-0\" (UID: \"163b57d1-c2a7-4939-8f82-77c01cd1721b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:49.471846 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.471820 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:49.804754 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.804574 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 21:40:49.808114 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:40:49.808085 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod163b57d1_c2a7_4939_8f82_77c01cd1721b.slice/crio-87c2bf8204c5e774df5822107323a199213936445cd2494ede84bfee8a47f8cd WatchSource:0}: Error finding container 87c2bf8204c5e774df5822107323a199213936445cd2494ede84bfee8a47f8cd: Status 404 returned error can't find the container with id 87c2bf8204c5e774df5822107323a199213936445cd2494ede84bfee8a47f8cd Oct 30 21:40:49.954415 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:49.954367 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b" path="/var/lib/kubelet/pods/1a5ccd5b-b287-4b3f-95cf-2d6d6c7c1f2b/volumes" Oct 30 21:40:50.079039 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:50.078969 6720 generic.go:356] "Generic (PLEG): container finished" podID="163b57d1-c2a7-4939-8f82-77c01cd1721b" containerID="f1597b5d248e7954a2a665bc9e57a3820b580fc64ce698729b11717839d2dc00" exitCode=0 Oct 30 21:40:50.079179 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:50.079064 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"163b57d1-c2a7-4939-8f82-77c01cd1721b","Type":"ContainerDied","Data":"f1597b5d248e7954a2a665bc9e57a3820b580fc64ce698729b11717839d2dc00"} Oct 30 21:40:50.079179 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:50.079146 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"163b57d1-c2a7-4939-8f82-77c01cd1721b","Type":"ContainerStarted","Data":"87c2bf8204c5e774df5822107323a199213936445cd2494ede84bfee8a47f8cd"} Oct 30 21:40:50.704775 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:50.704742 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ec3fe241-c753-4845-8b7a-0469c73e4c3b-service-ca\") pod \"console-6db5565879-ndj8n\" (UID: \"ec3fe241-c753-4845-8b7a-0469c73e4c3b\") " pod="openshift-console/console-6db5565879-ndj8n" Oct 30 21:40:50.704775 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:50.704780 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ec3fe241-c753-4845-8b7a-0469c73e4c3b-console-config\") pod \"console-6db5565879-ndj8n\" (UID: \"ec3fe241-c753-4845-8b7a-0469c73e4c3b\") " pod="openshift-console/console-6db5565879-ndj8n" Oct 30 21:40:50.705246 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:50.704819 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ec3fe241-c753-4845-8b7a-0469c73e4c3b-oauth-serving-cert\") pod \"console-6db5565879-ndj8n\" (UID: \"ec3fe241-c753-4845-8b7a-0469c73e4c3b\") " pod="openshift-console/console-6db5565879-ndj8n" Oct 30 21:40:50.705246 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:50.704875 6720 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered Oct 30 21:40:50.705246 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:50.704878 6720 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered Oct 30 21:40:50.705246 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:50.704919 6720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ec3fe241-c753-4845-8b7a-0469c73e4c3b-oauth-serving-cert podName:ec3fe241-c753-4845-8b7a-0469c73e4c3b nodeName:}" failed. No retries permitted until 2026-10-30 21:41:06.704907679 +0000 UTC m=+32.875872839 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/ec3fe241-c753-4845-8b7a-0469c73e4c3b-oauth-serving-cert") pod "console-6db5565879-ndj8n" (UID: "ec3fe241-c753-4845-8b7a-0469c73e4c3b") : object "openshift-console"/"oauth-serving-cert" not registered Oct 30 21:40:50.705246 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:50.704880 6720 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered Oct 30 21:40:50.705246 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:50.704932 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ec3fe241-c753-4845-8b7a-0469c73e4c3b-console-serving-cert\") pod \"console-6db5565879-ndj8n\" (UID: \"ec3fe241-c753-4845-8b7a-0469c73e4c3b\") " pod="openshift-console/console-6db5565879-ndj8n" Oct 30 21:40:50.705246 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:50.704948 6720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ec3fe241-c753-4845-8b7a-0469c73e4c3b-service-ca podName:ec3fe241-c753-4845-8b7a-0469c73e4c3b nodeName:}" failed. No retries permitted until 2026-10-30 21:41:06.704936548 +0000 UTC m=+32.875901708 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/ec3fe241-c753-4845-8b7a-0469c73e4c3b-service-ca") pod "console-6db5565879-ndj8n" (UID: "ec3fe241-c753-4845-8b7a-0469c73e4c3b") : object "openshift-console"/"service-ca" not registered Oct 30 21:40:50.705246 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:50.704965 6720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ec3fe241-c753-4845-8b7a-0469c73e4c3b-console-config podName:ec3fe241-c753-4845-8b7a-0469c73e4c3b nodeName:}" failed. No retries permitted until 2026-10-30 21:41:06.704959746 +0000 UTC m=+32.875924906 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/ec3fe241-c753-4845-8b7a-0469c73e4c3b-console-config") pod "console-6db5565879-ndj8n" (UID: "ec3fe241-c753-4845-8b7a-0469c73e4c3b") : object "openshift-console"/"console-config" not registered Oct 30 21:40:50.705246 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:50.705022 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ec3fe241-c753-4845-8b7a-0469c73e4c3b-console-oauth-config\") pod \"console-6db5565879-ndj8n\" (UID: \"ec3fe241-c753-4845-8b7a-0469c73e4c3b\") " pod="openshift-console/console-6db5565879-ndj8n" Oct 30 21:40:50.705246 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:50.705029 6720 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered Oct 30 21:40:50.705246 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:50.705087 6720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec3fe241-c753-4845-8b7a-0469c73e4c3b-console-serving-cert podName:ec3fe241-c753-4845-8b7a-0469c73e4c3b nodeName:}" failed. No retries permitted until 2026-10-30 21:41:06.705073315 +0000 UTC m=+32.876038481 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/ec3fe241-c753-4845-8b7a-0469c73e4c3b-console-serving-cert") pod "console-6db5565879-ndj8n" (UID: "ec3fe241-c753-4845-8b7a-0469c73e4c3b") : object "openshift-console"/"console-serving-cert" not registered Oct 30 21:40:50.705246 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:50.705119 6720 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered Oct 30 21:40:50.705246 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:50.705149 6720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec3fe241-c753-4845-8b7a-0469c73e4c3b-console-oauth-config podName:ec3fe241-c753-4845-8b7a-0469c73e4c3b nodeName:}" failed. No retries permitted until 2026-10-30 21:41:06.705137244 +0000 UTC m=+32.876102404 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/ec3fe241-c753-4845-8b7a-0469c73e4c3b-console-oauth-config") pod "console-6db5565879-ndj8n" (UID: "ec3fe241-c753-4845-8b7a-0469c73e4c3b") : object "openshift-console"/"console-oauth-config" not registered Oct 30 21:40:50.805464 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:50.805438 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rczmc\" (UniqueName: \"kubernetes.io/projected/ec3fe241-c753-4845-8b7a-0469c73e4c3b-kube-api-access-rczmc\") pod \"console-6db5565879-ndj8n\" (UID: \"ec3fe241-c753-4845-8b7a-0469c73e4c3b\") " pod="openshift-console/console-6db5565879-ndj8n" Oct 30 21:40:50.805631 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:50.805606 6720 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered Oct 30 21:40:50.805631 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:50.805626 6720 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered Oct 30 21:40:50.805742 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:50.805638 6720 projected.go:196] Error preparing data for projected volume kube-api-access-rczmc for pod openshift-console/console-6db5565879-ndj8n: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 21:40:50.805742 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:40:50.805696 6720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ec3fe241-c753-4845-8b7a-0469c73e4c3b-kube-api-access-rczmc podName:ec3fe241-c753-4845-8b7a-0469c73e4c3b nodeName:}" failed. No retries permitted until 2026-10-30 21:41:06.805679793 +0000 UTC m=+32.976644968 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-rczmc" (UniqueName: "kubernetes.io/projected/ec3fe241-c753-4845-8b7a-0469c73e4c3b-kube-api-access-rczmc") pod "console-6db5565879-ndj8n" (UID: "ec3fe241-c753-4845-8b7a-0469c73e4c3b") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered] Oct 30 21:40:51.087067 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:51.086985 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"163b57d1-c2a7-4939-8f82-77c01cd1721b","Type":"ContainerStarted","Data":"eb7567e8bd773bd6374122170cfcdb0b77b5593290b9002d6517bfeb3fe406b5"} Oct 30 21:40:51.087205 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:51.087098 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"163b57d1-c2a7-4939-8f82-77c01cd1721b","Type":"ContainerStarted","Data":"05e08db2cc17bdc413f41d14d518a124fd54d552756a5e30779dc357df71744d"} Oct 30 21:40:51.087205 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:51.087148 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"163b57d1-c2a7-4939-8f82-77c01cd1721b","Type":"ContainerStarted","Data":"b52a33caff164b8dbbaa56dd2d23418ba7683fa4cde5b0164122a5b8607b1a61"} Oct 30 21:40:51.087205 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:51.087186 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"163b57d1-c2a7-4939-8f82-77c01cd1721b","Type":"ContainerStarted","Data":"aa18e1990f9cb96519b576fbaba2653a80aef9f220224f1ab6c00c3e7f6e820c"} Oct 30 21:40:51.087340 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:51.087222 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"163b57d1-c2a7-4939-8f82-77c01cd1721b","Type":"ContainerStarted","Data":"2bd213cb7867c51002c620f4cad6df0ab9565a9841e436385f3445b26a61c388"} Oct 30 21:40:51.087340 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:51.087258 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"163b57d1-c2a7-4939-8f82-77c01cd1721b","Type":"ContainerStarted","Data":"c8939891c44e4bfdc39878d16e56cc20d40d3b68bd115916c030c50910148687"} Oct 30 21:40:51.126270 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:51.126223 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.1153985730000002 podStartE2EDuration="2.126210828s" totalImagesPullingTime="0s" totalInitContainerRuntime="10.812255ms" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:51.124650625 +0000 UTC m=+17.295615808" watchObservedRunningTime="2026-10-30 21:40:51.126210828 +0000 UTC m=+17.297176004" Oct 30 21:40:52.081355 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:52.081312 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:52.226210 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:52.226181 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:52.444237 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:52.444210 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:52.586600 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:52.586572 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:40:54.472218 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:54.472178 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:40:59.954501 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:40:59.954462 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-6db5565879-ndj8n" podUID="ec3fe241-c753-4845-8b7a-0469c73e4c3b" containerName="console" containerID="cri-o://2a6e9dd3dd9adf00dada2b6d30b6309408993f8f70e9000cfb148ba4d85a30f5" gracePeriod=15 Oct 30 21:41:00.120939 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:41:00.120902 6720 generic.go:356] "Generic (PLEG): container finished" podID="ec3fe241-c753-4845-8b7a-0469c73e4c3b" containerID="2a6e9dd3dd9adf00dada2b6d30b6309408993f8f70e9000cfb148ba4d85a30f5" exitCode=0 Oct 30 21:41:00.121097 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:41:00.120974 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6db5565879-ndj8n" event={"ID":"ec3fe241-c753-4845-8b7a-0469c73e4c3b","Type":"ContainerDied","Data":"2a6e9dd3dd9adf00dada2b6d30b6309408993f8f70e9000cfb148ba4d85a30f5"} Oct 30 21:41:00.197919 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:41:00.197894 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6db5565879-ndj8n" Oct 30 21:41:00.288971 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:41:00.288907 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec3fe241-c753-4845-8b7a-0469c73e4c3b-console-config\" (UniqueName: \"kubernetes.io/configmap/ec3fe241-c753-4845-8b7a-0469c73e4c3b-console-config\") pod \"ec3fe241-c753-4845-8b7a-0469c73e4c3b\" (UID: \"ec3fe241-c753-4845-8b7a-0469c73e4c3b\") " Oct 30 21:41:00.288971 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:41:00.288955 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec3fe241-c753-4845-8b7a-0469c73e4c3b-kube-api-access-rczmc\" (UniqueName: \"kubernetes.io/projected/ec3fe241-c753-4845-8b7a-0469c73e4c3b-kube-api-access-rczmc\") pod \"ec3fe241-c753-4845-8b7a-0469c73e4c3b\" (UID: \"ec3fe241-c753-4845-8b7a-0469c73e4c3b\") " Oct 30 21:41:00.289161 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:41:00.288977 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec3fe241-c753-4845-8b7a-0469c73e4c3b-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ec3fe241-c753-4845-8b7a-0469c73e4c3b-oauth-serving-cert\") pod \"ec3fe241-c753-4845-8b7a-0469c73e4c3b\" (UID: \"ec3fe241-c753-4845-8b7a-0469c73e4c3b\") " Oct 30 21:41:00.289161 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:41:00.289003 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec3fe241-c753-4845-8b7a-0469c73e4c3b-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec3fe241-c753-4845-8b7a-0469c73e4c3b-service-ca\") pod \"ec3fe241-c753-4845-8b7a-0469c73e4c3b\" (UID: \"ec3fe241-c753-4845-8b7a-0469c73e4c3b\") " Oct 30 21:41:00.289161 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:41:00.289047 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec3fe241-c753-4845-8b7a-0469c73e4c3b-tmp\" (UniqueName: \"kubernetes.io/empty-dir/ec3fe241-c753-4845-8b7a-0469c73e4c3b-tmp\") pod \"ec3fe241-c753-4845-8b7a-0469c73e4c3b\" (UID: \"ec3fe241-c753-4845-8b7a-0469c73e4c3b\") " Oct 30 21:41:00.289161 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:41:00.289102 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec3fe241-c753-4845-8b7a-0469c73e4c3b-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ec3fe241-c753-4845-8b7a-0469c73e4c3b-console-serving-cert\") pod \"ec3fe241-c753-4845-8b7a-0469c73e4c3b\" (UID: \"ec3fe241-c753-4845-8b7a-0469c73e4c3b\") " Oct 30 21:41:00.289358 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:41:00.289165 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec3fe241-c753-4845-8b7a-0469c73e4c3b-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ec3fe241-c753-4845-8b7a-0469c73e4c3b-console-oauth-config\") pod \"ec3fe241-c753-4845-8b7a-0469c73e4c3b\" (UID: \"ec3fe241-c753-4845-8b7a-0469c73e4c3b\") " Oct 30 21:41:00.289442 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:41:00.289346 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec3fe241-c753-4845-8b7a-0469c73e4c3b-console-config" pod "ec3fe241-c753-4845-8b7a-0469c73e4c3b" (UID: "ec3fe241-c753-4845-8b7a-0469c73e4c3b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:41:00.289567 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:41:00.289511 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec3fe241-c753-4845-8b7a-0469c73e4c3b-tmp" pod "ec3fe241-c753-4845-8b7a-0469c73e4c3b" (UID: "ec3fe241-c753-4845-8b7a-0469c73e4c3b"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:41:00.289567 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:41:00.289376 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec3fe241-c753-4845-8b7a-0469c73e4c3b-oauth-serving-cert" pod "ec3fe241-c753-4845-8b7a-0469c73e4c3b" (UID: "ec3fe241-c753-4845-8b7a-0469c73e4c3b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:41:00.289567 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:41:00.289538 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec3fe241-c753-4845-8b7a-0469c73e4c3b-service-ca" pod "ec3fe241-c753-4845-8b7a-0469c73e4c3b" (UID: "ec3fe241-c753-4845-8b7a-0469c73e4c3b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:41:00.289761 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:41:00.289642 6720 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ec3fe241-c753-4845-8b7a-0469c73e4c3b-tmp\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:41:00.289761 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:41:00.289662 6720 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ec3fe241-c753-4845-8b7a-0469c73e4c3b-console-config\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:41:00.289761 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:41:00.289675 6720 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ec3fe241-c753-4845-8b7a-0469c73e4c3b-oauth-serving-cert\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:41:00.289761 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:41:00.289683 6720 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ec3fe241-c753-4845-8b7a-0469c73e4c3b-service-ca\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:41:00.291544 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:41:00.291518 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec3fe241-c753-4845-8b7a-0469c73e4c3b-console-serving-cert" pod "ec3fe241-c753-4845-8b7a-0469c73e4c3b" (UID: "ec3fe241-c753-4845-8b7a-0469c73e4c3b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:41:00.291639 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:41:00.291556 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec3fe241-c753-4845-8b7a-0469c73e4c3b-console-oauth-config" pod "ec3fe241-c753-4845-8b7a-0469c73e4c3b" (UID: "ec3fe241-c753-4845-8b7a-0469c73e4c3b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:41:00.291639 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:41:00.291558 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec3fe241-c753-4845-8b7a-0469c73e4c3b-kube-api-access-rczmc" pod "ec3fe241-c753-4845-8b7a-0469c73e4c3b" (UID: "ec3fe241-c753-4845-8b7a-0469c73e4c3b"). InnerVolumeSpecName "kube-api-access-rczmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:41:00.390705 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:41:00.390678 6720 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ec3fe241-c753-4845-8b7a-0469c73e4c3b-console-oauth-config\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:41:00.390705 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:41:00.390700 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rczmc\" (UniqueName: \"kubernetes.io/projected/ec3fe241-c753-4845-8b7a-0469c73e4c3b-kube-api-access-rczmc\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:41:00.390705 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:41:00.390710 6720 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ec3fe241-c753-4845-8b7a-0469c73e4c3b-console-serving-cert\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:41:01.129567 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:41:01.129524 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6db5565879-ndj8n" event={"ID":"ec3fe241-c753-4845-8b7a-0469c73e4c3b","Type":"ContainerDied","Data":"6c704cae0f9a5285742a852fc15009b450a18501da7c1d9946d7526ad8eae067"} Oct 30 21:41:01.129567 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:41:01.129560 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6db5565879-ndj8n" Oct 30 21:41:01.130094 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:41:01.129584 6720 scope.go:122] "RemoveContainer" containerID="2a6e9dd3dd9adf00dada2b6d30b6309408993f8f70e9000cfb148ba4d85a30f5" Oct 30 21:41:01.153280 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:41:01.153255 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6db5565879-ndj8n"] Oct 30 21:41:01.159105 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:41:01.159084 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6db5565879-ndj8n"] Oct 30 21:41:01.953838 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:41:01.953750 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec3fe241-c753-4845-8b7a-0469c73e4c3b" path="/var/lib/kubelet/pods/ec3fe241-c753-4845-8b7a-0469c73e4c3b/volumes" Oct 30 21:41:03.302867 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:41:03.302838 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-jg4wr" Oct 30 21:41:03.778100 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:41:03.778063 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-75z8j" Oct 30 21:41:09.424977 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:41:09.424939 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-9h6ck" Oct 30 21:41:10.402929 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:41:10.402895 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 21:41:22.341835 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:41:22.341802 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:22.488835 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:41:22.488805 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:25.575427 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:41:25.575362 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:40.400802 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:41:40.400763 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 21:41:41.713313 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:41:41.713276 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:41.863415 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:41:41.863374 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:49.020626 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:41:49.020595 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:41:49.036756 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:41:49.036730 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:41:49.311372 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:41:49.311302 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 21:41:58.155496 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:41:58.155459 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:58.345898 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:41:58.345863 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:59.648394 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:41:59.648348 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:59.782404 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:41:59.782346 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:41:59.948640 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:41:59.948563 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:00.096492 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:42:00.096458 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:03.523906 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:42:03.523869 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:10.400761 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:42:10.400701 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 21:42:10.785142 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:42:10.785054 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:10.934521 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:42:10.934482 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:15.444938 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:42:15.444860 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:15.594146 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:42:15.594115 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:25.625899 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:42:25.625863 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:42:40.400969 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:42:40.400933 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 21:43:10.400923 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:43:10.400884 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 21:43:25.670787 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:43:25.670751 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:40.401017 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:43:40.400982 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 21:43:45.626019 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:43:45.625935 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:45.811441 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:43:45.811412 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:47.458827 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:43:47.458790 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:47.627858 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:43:47.627828 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:47.928659 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:43:47.928628 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:48.081135 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:43:48.081108 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:48.735820 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:43:48.735789 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:53.183273 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:43:53.183238 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:53.331282 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:43:53.331249 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:53.575578 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:43:53.575551 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:53.720254 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:43:53.720226 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:53.871133 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:43:53.871106 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:53.971235 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:43:53.971207 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cxgdb"] Oct 30 21:43:53.971687 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:43:53.971668 6720 cpu_manager.go:414] "containerMap: removing container" podUID="ec3fe241-c753-4845-8b7a-0469c73e4c3b" containerName="console" Oct 30 21:43:53.971797 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:43:53.971688 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec3fe241-c753-4845-8b7a-0469c73e4c3b" containerName="console" Oct 30 21:43:53.971797 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:43:53.971766 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec3fe241-c753-4845-8b7a-0469c73e4c3b" containerName="console" Oct 30 21:43:53.974861 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:43:53.974840 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cxgdb" Oct 30 21:43:53.980756 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:43:53.980733 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-hzztr\"" Oct 30 21:43:53.980848 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:43:53.980773 6720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 21:43:53.980848 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:43:53.980781 6720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 21:43:53.993471 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:43:53.993450 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cxgdb"] Oct 30 21:43:54.111425 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:43:54.111369 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1a93a094-fc39-4518-bd79-f5956f77b2e3-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cxgdb\" (UID: \"1a93a094-fc39-4518-bd79-f5956f77b2e3\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cxgdb" Oct 30 21:43:54.111579 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:43:54.111444 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lk4m\" (UniqueName: \"kubernetes.io/projected/1a93a094-fc39-4518-bd79-f5956f77b2e3-kube-api-access-8lk4m\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cxgdb\" (UID: \"1a93a094-fc39-4518-bd79-f5956f77b2e3\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cxgdb" Oct 30 21:43:54.111579 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:43:54.111560 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1a93a094-fc39-4518-bd79-f5956f77b2e3-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cxgdb\" (UID: \"1a93a094-fc39-4518-bd79-f5956f77b2e3\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cxgdb" Oct 30 21:43:54.212578 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:43:54.212501 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1a93a094-fc39-4518-bd79-f5956f77b2e3-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cxgdb\" (UID: \"1a93a094-fc39-4518-bd79-f5956f77b2e3\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cxgdb" Oct 30 21:43:54.212578 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:43:54.212562 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1a93a094-fc39-4518-bd79-f5956f77b2e3-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cxgdb\" (UID: \"1a93a094-fc39-4518-bd79-f5956f77b2e3\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cxgdb" Oct 30 21:43:54.213088 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:43:54.212583 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lk4m\" (UniqueName: \"kubernetes.io/projected/1a93a094-fc39-4518-bd79-f5956f77b2e3-kube-api-access-8lk4m\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cxgdb\" (UID: \"1a93a094-fc39-4518-bd79-f5956f77b2e3\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cxgdb" Oct 30 21:43:54.213088 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:43:54.212994 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1a93a094-fc39-4518-bd79-f5956f77b2e3-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cxgdb\" (UID: \"1a93a094-fc39-4518-bd79-f5956f77b2e3\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cxgdb" Oct 30 21:43:54.213088 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:43:54.212995 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1a93a094-fc39-4518-bd79-f5956f77b2e3-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cxgdb\" (UID: \"1a93a094-fc39-4518-bd79-f5956f77b2e3\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cxgdb" Oct 30 21:43:54.226225 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:43:54.226193 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lk4m\" (UniqueName: \"kubernetes.io/projected/1a93a094-fc39-4518-bd79-f5956f77b2e3-kube-api-access-8lk4m\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cxgdb\" (UID: \"1a93a094-fc39-4518-bd79-f5956f77b2e3\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cxgdb" Oct 30 21:43:54.291029 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:43:54.291009 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cxgdb" Oct 30 21:43:54.571665 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:43:54.571637 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cxgdb"] Oct 30 21:43:54.573136 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:43:54.573108 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a93a094_fc39_4518_bd79_f5956f77b2e3.slice/crio-843c69fd4e1846aab44375ecde20f1c8ff5e7eaefb7165aaa170e256ba5bec1b WatchSource:0}: Error finding container 843c69fd4e1846aab44375ecde20f1c8ff5e7eaefb7165aaa170e256ba5bec1b: Status 404 returned error can't find the container with id 843c69fd4e1846aab44375ecde20f1c8ff5e7eaefb7165aaa170e256ba5bec1b Oct 30 21:43:54.575194 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:43:54.575176 6720 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:43:54.720931 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:43:54.720896 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cxgdb" event={"ID":"1a93a094-fc39-4518-bd79-f5956f77b2e3","Type":"ContainerStarted","Data":"843c69fd4e1846aab44375ecde20f1c8ff5e7eaefb7165aaa170e256ba5bec1b"} Oct 30 21:43:56.283217 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:43:56.283184 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:43:59.948524 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:43:59.948493 6720 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a93a094_fc39_4518_bd79_f5956f77b2e3.slice/crio-f3f0363ff07cd4c293bea50386c8e99cdc0ed1e41232f02cff7ab7d592a60b8f.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a93a094_fc39_4518_bd79_f5956f77b2e3.slice/crio-conmon-f3f0363ff07cd4c293bea50386c8e99cdc0ed1e41232f02cff7ab7d592a60b8f.scope\": RecentStats: unable to find data in memory cache]" Oct 30 21:44:00.746633 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:00.746603 6720 generic.go:356] "Generic (PLEG): container finished" podID="1a93a094-fc39-4518-bd79-f5956f77b2e3" containerID="f3f0363ff07cd4c293bea50386c8e99cdc0ed1e41232f02cff7ab7d592a60b8f" exitCode=0 Oct 30 21:44:00.746822 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:00.746686 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cxgdb" event={"ID":"1a93a094-fc39-4518-bd79-f5956f77b2e3","Type":"ContainerDied","Data":"f3f0363ff07cd4c293bea50386c8e99cdc0ed1e41232f02cff7ab7d592a60b8f"} Oct 30 21:44:04.763000 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:04.762961 6720 generic.go:356] "Generic (PLEG): container finished" podID="1a93a094-fc39-4518-bd79-f5956f77b2e3" containerID="682cabc8e9aebce61ec76dfce4426068c0b7f0a0b4d7c20cc09edc1e05b389fb" exitCode=0 Oct 30 21:44:04.763000 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:04.762998 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cxgdb" event={"ID":"1a93a094-fc39-4518-bd79-f5956f77b2e3","Type":"ContainerDied","Data":"682cabc8e9aebce61ec76dfce4426068c0b7f0a0b4d7c20cc09edc1e05b389fb"} Oct 30 21:44:10.406431 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:44:10.406372 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 21:44:10.788039 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:10.787956 6720 generic.go:356] "Generic (PLEG): container finished" podID="1a93a094-fc39-4518-bd79-f5956f77b2e3" containerID="d5f3ca0eee9fc49e116ce9da335ad3dfe9fa7f8a1cfefc8c687c1f85f29bda57" exitCode=0 Oct 30 21:44:10.788039 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:10.788024 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cxgdb" event={"ID":"1a93a094-fc39-4518-bd79-f5956f77b2e3","Type":"ContainerDied","Data":"d5f3ca0eee9fc49e116ce9da335ad3dfe9fa7f8a1cfefc8c687c1f85f29bda57"} Oct 30 21:44:11.914454 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:11.914430 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cxgdb" Oct 30 21:44:11.966567 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:11.966540 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a93a094-fc39-4518-bd79-f5956f77b2e3-bundle\" (UniqueName: \"kubernetes.io/empty-dir/1a93a094-fc39-4518-bd79-f5956f77b2e3-bundle\") pod \"1a93a094-fc39-4518-bd79-f5956f77b2e3\" (UID: \"1a93a094-fc39-4518-bd79-f5956f77b2e3\") " Oct 30 21:44:11.966696 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:11.966619 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a93a094-fc39-4518-bd79-f5956f77b2e3-kube-api-access-8lk4m\" (UniqueName: \"kubernetes.io/projected/1a93a094-fc39-4518-bd79-f5956f77b2e3-kube-api-access-8lk4m\") pod \"1a93a094-fc39-4518-bd79-f5956f77b2e3\" (UID: \"1a93a094-fc39-4518-bd79-f5956f77b2e3\") " Oct 30 21:44:11.966696 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:11.966653 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a93a094-fc39-4518-bd79-f5956f77b2e3-util\" (UniqueName: \"kubernetes.io/empty-dir/1a93a094-fc39-4518-bd79-f5956f77b2e3-util\") pod \"1a93a094-fc39-4518-bd79-f5956f77b2e3\" (UID: \"1a93a094-fc39-4518-bd79-f5956f77b2e3\") " Oct 30 21:44:11.967007 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:11.966984 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a93a094-fc39-4518-bd79-f5956f77b2e3-bundle" pod "1a93a094-fc39-4518-bd79-f5956f77b2e3" (UID: "1a93a094-fc39-4518-bd79-f5956f77b2e3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:44:11.968927 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:11.968900 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a93a094-fc39-4518-bd79-f5956f77b2e3-kube-api-access-8lk4m" pod "1a93a094-fc39-4518-bd79-f5956f77b2e3" (UID: "1a93a094-fc39-4518-bd79-f5956f77b2e3"). InnerVolumeSpecName "kube-api-access-8lk4m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:44:11.970583 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:11.970560 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a93a094-fc39-4518-bd79-f5956f77b2e3-util" pod "1a93a094-fc39-4518-bd79-f5956f77b2e3" (UID: "1a93a094-fc39-4518-bd79-f5956f77b2e3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:44:12.067246 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:12.067191 6720 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1a93a094-fc39-4518-bd79-f5956f77b2e3-bundle\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:44:12.067246 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:12.067212 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lk4m\" (UniqueName: \"kubernetes.io/projected/1a93a094-fc39-4518-bd79-f5956f77b2e3-kube-api-access-8lk4m\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:44:12.067246 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:12.067222 6720 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1a93a094-fc39-4518-bd79-f5956f77b2e3-util\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:44:12.797195 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:12.797169 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cxgdb" Oct 30 21:44:12.797358 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:12.797169 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644cxgdb" event={"ID":"1a93a094-fc39-4518-bd79-f5956f77b2e3","Type":"ContainerDied","Data":"843c69fd4e1846aab44375ecde20f1c8ff5e7eaefb7165aaa170e256ba5bec1b"} Oct 30 21:44:12.797358 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:12.797285 6720 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="843c69fd4e1846aab44375ecde20f1c8ff5e7eaefb7165aaa170e256ba5bec1b" Oct 30 21:44:14.223417 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:14.223373 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:14.418751 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:14.418709 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:15.521008 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:15.520976 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-7j9st"] Oct 30 21:44:15.521430 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:15.521284 6720 cpu_manager.go:414] "containerMap: removing container" podUID="1a93a094-fc39-4518-bd79-f5956f77b2e3" containerName="util" Oct 30 21:44:15.521430 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:15.521298 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a93a094-fc39-4518-bd79-f5956f77b2e3" containerName="util" Oct 30 21:44:15.521430 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:15.521321 6720 cpu_manager.go:414] "containerMap: removing container" podUID="1a93a094-fc39-4518-bd79-f5956f77b2e3" containerName="extract" Oct 30 21:44:15.521430 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:15.521326 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a93a094-fc39-4518-bd79-f5956f77b2e3" containerName="extract" Oct 30 21:44:15.521430 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:15.521335 6720 cpu_manager.go:414] "containerMap: removing container" podUID="1a93a094-fc39-4518-bd79-f5956f77b2e3" containerName="pull" Oct 30 21:44:15.521430 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:15.521341 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a93a094-fc39-4518-bd79-f5956f77b2e3" containerName="pull" Oct 30 21:44:15.521430 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:15.521410 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a93a094-fc39-4518-bd79-f5956f77b2e3" containerName="extract" Oct 30 21:44:15.525936 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:15.525918 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-7j9st" Oct 30 21:44:15.533060 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:15.533036 6720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:44:15.533150 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:15.533084 6720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Oct 30 21:44:15.534656 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:15.534637 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-vswvj\"" Oct 30 21:44:15.536849 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:15.536831 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-7j9st"] Oct 30 21:44:15.594580 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:15.594550 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rcsz\" (UniqueName: \"kubernetes.io/projected/7f44945c-60b7-4af5-8d62-ac80c7939867-kube-api-access-6rcsz\") pod \"jobset-operator-674cd59dc5-7j9st\" (UID: \"7f44945c-60b7-4af5-8d62-ac80c7939867\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-7j9st" Oct 30 21:44:15.594692 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:15.594609 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7f44945c-60b7-4af5-8d62-ac80c7939867-tmp\") pod \"jobset-operator-674cd59dc5-7j9st\" (UID: \"7f44945c-60b7-4af5-8d62-ac80c7939867\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-7j9st" Oct 30 21:44:15.695984 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:15.695957 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7f44945c-60b7-4af5-8d62-ac80c7939867-tmp\") pod \"jobset-operator-674cd59dc5-7j9st\" (UID: \"7f44945c-60b7-4af5-8d62-ac80c7939867\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-7j9st" Oct 30 21:44:15.696106 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:15.696004 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rcsz\" (UniqueName: \"kubernetes.io/projected/7f44945c-60b7-4af5-8d62-ac80c7939867-kube-api-access-6rcsz\") pod \"jobset-operator-674cd59dc5-7j9st\" (UID: \"7f44945c-60b7-4af5-8d62-ac80c7939867\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-7j9st" Oct 30 21:44:15.696319 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:15.696301 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7f44945c-60b7-4af5-8d62-ac80c7939867-tmp\") pod \"jobset-operator-674cd59dc5-7j9st\" (UID: \"7f44945c-60b7-4af5-8d62-ac80c7939867\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-7j9st" Oct 30 21:44:15.708792 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:15.708766 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rcsz\" (UniqueName: \"kubernetes.io/projected/7f44945c-60b7-4af5-8d62-ac80c7939867-kube-api-access-6rcsz\") pod \"jobset-operator-674cd59dc5-7j9st\" (UID: \"7f44945c-60b7-4af5-8d62-ac80c7939867\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-7j9st" Oct 30 21:44:15.854065 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:15.854038 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-7j9st" Oct 30 21:44:16.132357 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:16.132242 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-7j9st"] Oct 30 21:44:16.135671 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:44:16.135648 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7f44945c_60b7_4af5_8d62_ac80c7939867.slice/crio-9591a3de792b5179358e45c57d927a85bb61099fc191051ad81748bf20f9c124 WatchSource:0}: Error finding container 9591a3de792b5179358e45c57d927a85bb61099fc191051ad81748bf20f9c124: Status 404 returned error can't find the container with id 9591a3de792b5179358e45c57d927a85bb61099fc191051ad81748bf20f9c124 Oct 30 21:44:16.812534 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:16.812493 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-7j9st" event={"ID":"7f44945c-60b7-4af5-8d62-ac80c7939867","Type":"ContainerStarted","Data":"9591a3de792b5179358e45c57d927a85bb61099fc191051ad81748bf20f9c124"} Oct 30 21:44:19.825669 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:19.825636 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-7j9st" event={"ID":"7f44945c-60b7-4af5-8d62-ac80c7939867","Type":"ContainerStarted","Data":"813d737d33bd8947527dee3b6b299bce92e8cb9007d9d65cd4a162fe6c9c8a7a"} Oct 30 21:44:19.848211 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:19.848136 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-7j9st" podStartSLOduration=1.8808440910000002 podStartE2EDuration="4.848118878s" totalImagesPullingTime="2.967274787s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:19.848048063 +0000 UTC m=+226.019013247" watchObservedRunningTime="2026-10-30 21:44:19.848118878 +0000 UTC m=+226.019084056" Oct 30 21:44:24.141294 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:24.141256 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:24.297792 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:24.297756 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:25.191255 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:25.191227 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:25.289097 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:25.289065 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5cb74c89f6-rkl57"] Oct 30 21:44:25.291247 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:25.291230 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cb74c89f6-rkl57" Oct 30 21:44:25.333421 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:25.333378 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5cb74c89f6-rkl57"] Oct 30 21:44:25.373890 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:25.373863 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2chvq\" (UniqueName: \"kubernetes.io/projected/8e255872-4214-4bee-9d7f-bb7f135c3bf5-kube-api-access-2chvq\") pod \"opendatahub-operator-controller-manager-5cb74c89f6-rkl57\" (UID: \"8e255872-4214-4bee-9d7f-bb7f135c3bf5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cb74c89f6-rkl57" Oct 30 21:44:25.374029 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:25.373916 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8e255872-4214-4bee-9d7f-bb7f135c3bf5-metrics-cert\") pod \"opendatahub-operator-controller-manager-5cb74c89f6-rkl57\" (UID: \"8e255872-4214-4bee-9d7f-bb7f135c3bf5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cb74c89f6-rkl57" Oct 30 21:44:25.374029 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:25.373989 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8e255872-4214-4bee-9d7f-bb7f135c3bf5-tmp\") pod \"opendatahub-operator-controller-manager-5cb74c89f6-rkl57\" (UID: \"8e255872-4214-4bee-9d7f-bb7f135c3bf5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cb74c89f6-rkl57" Oct 30 21:44:25.374117 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:25.374033 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8e255872-4214-4bee-9d7f-bb7f135c3bf5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5cb74c89f6-rkl57\" (UID: \"8e255872-4214-4bee-9d7f-bb7f135c3bf5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cb74c89f6-rkl57" Oct 30 21:44:25.374117 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:25.374094 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8e255872-4214-4bee-9d7f-bb7f135c3bf5-webhook-cert\") pod \"opendatahub-operator-controller-manager-5cb74c89f6-rkl57\" (UID: \"8e255872-4214-4bee-9d7f-bb7f135c3bf5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cb74c89f6-rkl57" Oct 30 21:44:25.374184 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:25.374127 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8e255872-4214-4bee-9d7f-bb7f135c3bf5-manifests\") pod \"opendatahub-operator-controller-manager-5cb74c89f6-rkl57\" (UID: \"8e255872-4214-4bee-9d7f-bb7f135c3bf5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cb74c89f6-rkl57" Oct 30 21:44:25.378340 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:25.378316 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 21:44:25.378340 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:25.378329 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 21:44:25.378516 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:25.378440 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-wwlz7\"" Oct 30 21:44:25.378516 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:25.378502 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 21:44:25.381059 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:25.381045 6720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 21:44:25.381139 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:25.381064 6720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 21:44:25.475217 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:25.475135 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8e255872-4214-4bee-9d7f-bb7f135c3bf5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5cb74c89f6-rkl57\" (UID: \"8e255872-4214-4bee-9d7f-bb7f135c3bf5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cb74c89f6-rkl57" Oct 30 21:44:25.475217 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:25.475167 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8e255872-4214-4bee-9d7f-bb7f135c3bf5-webhook-cert\") pod \"opendatahub-operator-controller-manager-5cb74c89f6-rkl57\" (UID: \"8e255872-4214-4bee-9d7f-bb7f135c3bf5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cb74c89f6-rkl57" Oct 30 21:44:25.475217 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:25.475196 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8e255872-4214-4bee-9d7f-bb7f135c3bf5-manifests\") pod \"opendatahub-operator-controller-manager-5cb74c89f6-rkl57\" (UID: \"8e255872-4214-4bee-9d7f-bb7f135c3bf5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cb74c89f6-rkl57" Oct 30 21:44:25.475538 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:25.475254 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2chvq\" (UniqueName: \"kubernetes.io/projected/8e255872-4214-4bee-9d7f-bb7f135c3bf5-kube-api-access-2chvq\") pod \"opendatahub-operator-controller-manager-5cb74c89f6-rkl57\" (UID: \"8e255872-4214-4bee-9d7f-bb7f135c3bf5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cb74c89f6-rkl57" Oct 30 21:44:25.475538 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:25.475351 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8e255872-4214-4bee-9d7f-bb7f135c3bf5-metrics-cert\") pod \"opendatahub-operator-controller-manager-5cb74c89f6-rkl57\" (UID: \"8e255872-4214-4bee-9d7f-bb7f135c3bf5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cb74c89f6-rkl57" Oct 30 21:44:25.475538 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:25.475415 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8e255872-4214-4bee-9d7f-bb7f135c3bf5-tmp\") pod \"opendatahub-operator-controller-manager-5cb74c89f6-rkl57\" (UID: \"8e255872-4214-4bee-9d7f-bb7f135c3bf5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cb74c89f6-rkl57" Oct 30 21:44:25.475705 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:25.475594 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8e255872-4214-4bee-9d7f-bb7f135c3bf5-manifests\") pod \"opendatahub-operator-controller-manager-5cb74c89f6-rkl57\" (UID: \"8e255872-4214-4bee-9d7f-bb7f135c3bf5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cb74c89f6-rkl57" Oct 30 21:44:25.477915 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:25.477892 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8e255872-4214-4bee-9d7f-bb7f135c3bf5-tmp\") pod \"opendatahub-operator-controller-manager-5cb74c89f6-rkl57\" (UID: \"8e255872-4214-4bee-9d7f-bb7f135c3bf5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cb74c89f6-rkl57" Oct 30 21:44:25.478017 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:25.477925 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8e255872-4214-4bee-9d7f-bb7f135c3bf5-webhook-cert\") pod \"opendatahub-operator-controller-manager-5cb74c89f6-rkl57\" (UID: \"8e255872-4214-4bee-9d7f-bb7f135c3bf5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cb74c89f6-rkl57" Oct 30 21:44:25.478137 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:25.478113 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8e255872-4214-4bee-9d7f-bb7f135c3bf5-metrics-cert\") pod \"opendatahub-operator-controller-manager-5cb74c89f6-rkl57\" (UID: \"8e255872-4214-4bee-9d7f-bb7f135c3bf5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cb74c89f6-rkl57" Oct 30 21:44:25.478173 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:25.478117 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8e255872-4214-4bee-9d7f-bb7f135c3bf5-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5cb74c89f6-rkl57\" (UID: \"8e255872-4214-4bee-9d7f-bb7f135c3bf5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cb74c89f6-rkl57" Oct 30 21:44:25.488134 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:25.488110 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2chvq\" (UniqueName: \"kubernetes.io/projected/8e255872-4214-4bee-9d7f-bb7f135c3bf5-kube-api-access-2chvq\") pod \"opendatahub-operator-controller-manager-5cb74c89f6-rkl57\" (UID: \"8e255872-4214-4bee-9d7f-bb7f135c3bf5\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cb74c89f6-rkl57" Oct 30 21:44:25.502414 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:25.502378 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:25.603716 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:25.603691 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cb74c89f6-rkl57" Oct 30 21:44:25.650147 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:25.650115 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:25.825836 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:25.825764 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:25.902669 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:25.902533 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5cb74c89f6-rkl57"] Oct 30 21:44:25.905706 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:44:25.905681 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e255872_4214_4bee_9d7f_bb7f135c3bf5.slice/crio-bbdf86c55264d3628297f69b5f8ef8620d1f2eacb93d1e99e615430f6aabcebe WatchSource:0}: Error finding container bbdf86c55264d3628297f69b5f8ef8620d1f2eacb93d1e99e615430f6aabcebe: Status 404 returned error can't find the container with id bbdf86c55264d3628297f69b5f8ef8620d1f2eacb93d1e99e615430f6aabcebe Oct 30 21:44:26.858653 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:26.858615 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cb74c89f6-rkl57" event={"ID":"8e255872-4214-4bee-9d7f-bb7f135c3bf5","Type":"ContainerStarted","Data":"bbdf86c55264d3628297f69b5f8ef8620d1f2eacb93d1e99e615430f6aabcebe"} Oct 30 21:44:27.772931 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:27.772898 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:28.822595 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:28.822557 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:29.457409 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:29.457349 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:29.872457 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:29.872425 6720 generic.go:356] "Generic (PLEG): container finished" podID="8e255872-4214-4bee-9d7f-bb7f135c3bf5" containerID="df4cb6f05b9ef0ed39f5719954369cba64027a2e90838b52a400ffd6e5cfc46b" exitCode=0 Oct 30 21:44:29.872826 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:29.872511 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cb74c89f6-rkl57" event={"ID":"8e255872-4214-4bee-9d7f-bb7f135c3bf5","Type":"ContainerDied","Data":"df4cb6f05b9ef0ed39f5719954369cba64027a2e90838b52a400ffd6e5cfc46b"} Oct 30 21:44:30.852547 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:30.852512 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:30.878559 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:30.878532 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cb74c89f6-rkl57" event={"ID":"8e255872-4214-4bee-9d7f-bb7f135c3bf5","Type":"ContainerStarted","Data":"82851d6530b0e0a4e0abc88277b828ba23bf52299e0bcf16a883f3292ff14b81"} Oct 30 21:44:30.878921 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:30.878669 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cb74c89f6-rkl57" Oct 30 21:44:30.950704 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:30.949688 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cb74c89f6-rkl57" podStartSLOduration=2.101065814 podStartE2EDuration="5.949673787s" totalImagesPullingTime="3.383037655s" totalInitContainerRuntime="465.570318ms" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:25 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:30.94800465 +0000 UTC m=+237.118969850" watchObservedRunningTime="2026-10-30 21:44:30.949673787 +0000 UTC m=+237.120639019" Oct 30 21:44:32.754812 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:32.754776 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:32.930036 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:32.930009 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:33.683944 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:33.683888 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:33.947913 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:33.947848 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:35.110270 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:35.110227 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:35.317891 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:35.317860 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:36.012890 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:36.012858 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-79bch"] Oct 30 21:44:36.018916 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:36.018896 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-79bch" Oct 30 21:44:36.023821 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:36.023806 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-zwf2q\"" Oct 30 21:44:36.023898 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:36.023881 6720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 30 21:44:36.023898 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:36.023893 6720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 30 21:44:36.034973 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:36.034949 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-79bch"] Oct 30 21:44:36.069401 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:36.069362 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hkns\" (UniqueName: \"kubernetes.io/projected/7124e1e3-a3b7-4806-8a4a-609f9e2c0ae5-kube-api-access-4hkns\") pod \"cert-manager-webhook-5dbbbf75cc-79bch\" (UID: \"7124e1e3-a3b7-4806-8a4a-609f9e2c0ae5\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-79bch" Oct 30 21:44:36.069526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:36.069481 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7124e1e3-a3b7-4806-8a4a-609f9e2c0ae5-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-79bch\" (UID: \"7124e1e3-a3b7-4806-8a4a-609f9e2c0ae5\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-79bch" Oct 30 21:44:36.170009 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:36.169976 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hkns\" (UniqueName: \"kubernetes.io/projected/7124e1e3-a3b7-4806-8a4a-609f9e2c0ae5-kube-api-access-4hkns\") pod \"cert-manager-webhook-5dbbbf75cc-79bch\" (UID: \"7124e1e3-a3b7-4806-8a4a-609f9e2c0ae5\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-79bch" Oct 30 21:44:36.170361 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:36.170028 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7124e1e3-a3b7-4806-8a4a-609f9e2c0ae5-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-79bch\" (UID: \"7124e1e3-a3b7-4806-8a4a-609f9e2c0ae5\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-79bch" Oct 30 21:44:36.188445 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:36.188415 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7124e1e3-a3b7-4806-8a4a-609f9e2c0ae5-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-79bch\" (UID: \"7124e1e3-a3b7-4806-8a4a-609f9e2c0ae5\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-79bch" Oct 30 21:44:36.188561 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:36.188546 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hkns\" (UniqueName: \"kubernetes.io/projected/7124e1e3-a3b7-4806-8a4a-609f9e2c0ae5-kube-api-access-4hkns\") pod \"cert-manager-webhook-5dbbbf75cc-79bch\" (UID: \"7124e1e3-a3b7-4806-8a4a-609f9e2c0ae5\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-79bch" Oct 30 21:44:36.330334 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:36.330242 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-79bch" Oct 30 21:44:36.611231 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:36.611198 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-79bch"] Oct 30 21:44:36.614531 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:44:36.614505 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7124e1e3_a3b7_4806_8a4a_609f9e2c0ae5.slice/crio-11efec3159ca0068ed29504962cc4a86f2ef8fb83b7006f5e78de937f309fb80 WatchSource:0}: Error finding container 11efec3159ca0068ed29504962cc4a86f2ef8fb83b7006f5e78de937f309fb80: Status 404 returned error can't find the container with id 11efec3159ca0068ed29504962cc4a86f2ef8fb83b7006f5e78de937f309fb80 Oct 30 21:44:36.903736 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:36.903655 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-79bch" event={"ID":"7124e1e3-a3b7-4806-8a4a-609f9e2c0ae5","Type":"ContainerStarted","Data":"11efec3159ca0068ed29504962cc4a86f2ef8fb83b7006f5e78de937f309fb80"} Oct 30 21:44:37.391856 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:37.391824 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:37.585082 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:37.585041 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:38.150656 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:38.150625 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:40.401400 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:44:40.401346 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 21:44:40.922049 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:40.922016 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-79bch" event={"ID":"7124e1e3-a3b7-4806-8a4a-609f9e2c0ae5","Type":"ContainerStarted","Data":"eaf86340de43897bd7fbe1fab4d42c799d53c1af3b110bcae0f892c1067294c2"} Oct 30 21:44:40.922214 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:40.922132 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-79bch" Oct 30 21:44:40.942778 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:40.942735 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-79bch" podStartSLOduration=2.650760144 podStartE2EDuration="5.942718572s" totalImagesPullingTime="3.291958428s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:40.942179542 +0000 UTC m=+247.113144724" watchObservedRunningTime="2026-10-30 21:44:40.942718572 +0000 UTC m=+247.113683753" Oct 30 21:44:41.885715 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:41.885685 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5cb74c89f6-rkl57" Oct 30 21:44:42.350971 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:42.350891 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:42.997084 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:42.997049 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:43.213094 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:43.213059 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:43.213822 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:43.213796 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-d72dl"] Oct 30 21:44:43.217851 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:43.217835 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-d72dl" Oct 30 21:44:43.225466 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:43.225447 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-nrpxx\"" Oct 30 21:44:43.229117 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:43.229099 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-d72dl"] Oct 30 21:44:43.337603 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:43.337542 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3d2476fa-c9c2-4d0d-9d74-3616ff8d9369-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-d72dl\" (UID: \"3d2476fa-c9c2-4d0d-9d74-3616ff8d9369\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-d72dl" Oct 30 21:44:43.337603 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:43.337587 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4gwr\" (UniqueName: \"kubernetes.io/projected/3d2476fa-c9c2-4d0d-9d74-3616ff8d9369-kube-api-access-f4gwr\") pod \"cert-manager-cainjector-5c57bbd4fc-d72dl\" (UID: \"3d2476fa-c9c2-4d0d-9d74-3616ff8d9369\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-d72dl" Oct 30 21:44:43.438108 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:43.438076 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3d2476fa-c9c2-4d0d-9d74-3616ff8d9369-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-d72dl\" (UID: \"3d2476fa-c9c2-4d0d-9d74-3616ff8d9369\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-d72dl" Oct 30 21:44:43.438268 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:43.438125 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4gwr\" (UniqueName: \"kubernetes.io/projected/3d2476fa-c9c2-4d0d-9d74-3616ff8d9369-kube-api-access-f4gwr\") pod \"cert-manager-cainjector-5c57bbd4fc-d72dl\" (UID: \"3d2476fa-c9c2-4d0d-9d74-3616ff8d9369\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-d72dl" Oct 30 21:44:43.455747 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:43.455712 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3d2476fa-c9c2-4d0d-9d74-3616ff8d9369-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-d72dl\" (UID: \"3d2476fa-c9c2-4d0d-9d74-3616ff8d9369\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-d72dl" Oct 30 21:44:43.455885 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:43.455869 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4gwr\" (UniqueName: \"kubernetes.io/projected/3d2476fa-c9c2-4d0d-9d74-3616ff8d9369-kube-api-access-f4gwr\") pod \"cert-manager-cainjector-5c57bbd4fc-d72dl\" (UID: \"3d2476fa-c9c2-4d0d-9d74-3616ff8d9369\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-d72dl" Oct 30 21:44:43.530495 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:43.530456 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-d72dl" Oct 30 21:44:43.805893 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:43.805864 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-d72dl"] Oct 30 21:44:43.809618 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:44:43.809588 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3d2476fa_c9c2_4d0d_9d74_3616ff8d9369.slice/crio-181132b65f04f9abe2d65f83bdb695ee723b6b862094c6b9e4896c5e973396b9 WatchSource:0}: Error finding container 181132b65f04f9abe2d65f83bdb695ee723b6b862094c6b9e4896c5e973396b9: Status 404 returned error can't find the container with id 181132b65f04f9abe2d65f83bdb695ee723b6b862094c6b9e4896c5e973396b9 Oct 30 21:44:43.935801 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:43.935767 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-d72dl" event={"ID":"3d2476fa-c9c2-4d0d-9d74-3616ff8d9369","Type":"ContainerStarted","Data":"1ca5b8747e7d5e1f87a961a6d1530faeeb6a009b110f8ac65321ca4e3c6119c2"} Oct 30 21:44:43.935974 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:43.935807 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-d72dl" event={"ID":"3d2476fa-c9c2-4d0d-9d74-3616ff8d9369","Type":"ContainerStarted","Data":"181132b65f04f9abe2d65f83bdb695ee723b6b862094c6b9e4896c5e973396b9"} Oct 30 21:44:44.035910 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:44.035857 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-d72dl" podStartSLOduration=1.03584465 podStartE2EDuration="1.03584465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:44.033627265 +0000 UTC m=+250.204592465" watchObservedRunningTime="2026-10-30 21:44:44.03584465 +0000 UTC m=+250.206809829" Oct 30 21:44:44.966213 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:44.966179 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:45.238839 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:45.238749 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:46.882565 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:46.882532 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:46.929033 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:46.929010 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-79bch" Oct 30 21:44:47.122077 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:47.122043 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:47.940552 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:47.940516 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:48.757405 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:48.757358 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:49.374764 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:49.374730 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:52.223236 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:52.223200 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:52.403926 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:52.403894 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:53.947343 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:53.947310 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:54.992516 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:54.992483 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:55.318787 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:55.318693 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-x5rxl"] Oct 30 21:44:55.323642 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:55.323625 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-x5rxl" Oct 30 21:44:55.333496 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:55.333473 6720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 21:44:55.334873 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:55.334858 6720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 21:44:55.334939 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:55.334934 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-krwm5\"" Oct 30 21:44:55.337374 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:55.337354 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-x5rxl"] Oct 30 21:44:55.440527 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:55.440500 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpwt4\" (UniqueName: \"kubernetes.io/projected/1bb7e9b5-60a0-4cf9-a5ab-2e5e76d077aa-kube-api-access-rpwt4\") pod \"limitador-operator-controller-manager-5778bb4449-x5rxl\" (UID: \"1bb7e9b5-60a0-4cf9-a5ab-2e5e76d077aa\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-x5rxl" Oct 30 21:44:55.541681 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:55.541653 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpwt4\" (UniqueName: \"kubernetes.io/projected/1bb7e9b5-60a0-4cf9-a5ab-2e5e76d077aa-kube-api-access-rpwt4\") pod \"limitador-operator-controller-manager-5778bb4449-x5rxl\" (UID: \"1bb7e9b5-60a0-4cf9-a5ab-2e5e76d077aa\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-x5rxl" Oct 30 21:44:55.557326 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:55.557302 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpwt4\" (UniqueName: \"kubernetes.io/projected/1bb7e9b5-60a0-4cf9-a5ab-2e5e76d077aa-kube-api-access-rpwt4\") pod \"limitador-operator-controller-manager-5778bb4449-x5rxl\" (UID: \"1bb7e9b5-60a0-4cf9-a5ab-2e5e76d077aa\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-x5rxl" Oct 30 21:44:55.637755 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:55.637731 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-x5rxl" Oct 30 21:44:55.917289 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:55.917213 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-x5rxl"] Oct 30 21:44:55.921425 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:44:55.921394 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bb7e9b5_60a0_4cf9_a5ab_2e5e76d077aa.slice/crio-3bbfa51894de37e73f2512f54be812171481d718ae3407d5e00b5b68f116c019 WatchSource:0}: Error finding container 3bbfa51894de37e73f2512f54be812171481d718ae3407d5e00b5b68f116c019: Status 404 returned error can't find the container with id 3bbfa51894de37e73f2512f54be812171481d718ae3407d5e00b5b68f116c019 Oct 30 21:44:55.982660 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:55.982629 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-x5rxl" event={"ID":"1bb7e9b5-60a0-4cf9-a5ab-2e5e76d077aa","Type":"ContainerStarted","Data":"3bbfa51894de37e73f2512f54be812171481d718ae3407d5e00b5b68f116c019"} Oct 30 21:44:56.985243 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:56.985198 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:57.142400 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:57.142349 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:58.476973 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:58.476944 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:58.680439 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:58.680401 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:58.720424 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:58.720368 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-78559bccc4-9vqg5"] Oct 30 21:44:58.724311 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:58.724286 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-78559bccc4-9vqg5" Oct 30 21:44:58.735522 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:58.735466 6720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 21:44:58.735522 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:58.735496 6720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 21:44:58.737527 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:58.737496 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-sz7sq\"" Oct 30 21:44:58.738525 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:58.738474 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 21:44:58.739311 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:58.739244 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-78559bccc4-9vqg5"] Oct 30 21:44:58.768911 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:58.768865 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 21:44:58.774630 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:58.774603 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0271c996-234d-4291-aa7a-77d43569c247-tmp\") pod \"kube-auth-proxy-78559bccc4-9vqg5\" (UID: \"0271c996-234d-4291-aa7a-77d43569c247\") " pod="openshift-ingress/kube-auth-proxy-78559bccc4-9vqg5" Oct 30 21:44:58.774787 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:58.774652 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8p59h\" (UniqueName: \"kubernetes.io/projected/0271c996-234d-4291-aa7a-77d43569c247-kube-api-access-8p59h\") pod \"kube-auth-proxy-78559bccc4-9vqg5\" (UID: \"0271c996-234d-4291-aa7a-77d43569c247\") " pod="openshift-ingress/kube-auth-proxy-78559bccc4-9vqg5" Oct 30 21:44:58.774787 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:58.774691 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0271c996-234d-4291-aa7a-77d43569c247-tls-certs\") pod \"kube-auth-proxy-78559bccc4-9vqg5\" (UID: \"0271c996-234d-4291-aa7a-77d43569c247\") " pod="openshift-ingress/kube-auth-proxy-78559bccc4-9vqg5" Oct 30 21:44:58.875199 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:58.875162 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0271c996-234d-4291-aa7a-77d43569c247-tmp\") pod \"kube-auth-proxy-78559bccc4-9vqg5\" (UID: \"0271c996-234d-4291-aa7a-77d43569c247\") " pod="openshift-ingress/kube-auth-proxy-78559bccc4-9vqg5" Oct 30 21:44:58.875368 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:58.875212 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8p59h\" (UniqueName: \"kubernetes.io/projected/0271c996-234d-4291-aa7a-77d43569c247-kube-api-access-8p59h\") pod \"kube-auth-proxy-78559bccc4-9vqg5\" (UID: \"0271c996-234d-4291-aa7a-77d43569c247\") " pod="openshift-ingress/kube-auth-proxy-78559bccc4-9vqg5" Oct 30 21:44:58.875368 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:58.875251 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0271c996-234d-4291-aa7a-77d43569c247-tls-certs\") pod \"kube-auth-proxy-78559bccc4-9vqg5\" (UID: \"0271c996-234d-4291-aa7a-77d43569c247\") " pod="openshift-ingress/kube-auth-proxy-78559bccc4-9vqg5" Oct 30 21:44:58.877518 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:58.877483 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0271c996-234d-4291-aa7a-77d43569c247-tmp\") pod \"kube-auth-proxy-78559bccc4-9vqg5\" (UID: \"0271c996-234d-4291-aa7a-77d43569c247\") " pod="openshift-ingress/kube-auth-proxy-78559bccc4-9vqg5" Oct 30 21:44:58.877719 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:58.877700 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0271c996-234d-4291-aa7a-77d43569c247-tls-certs\") pod \"kube-auth-proxy-78559bccc4-9vqg5\" (UID: \"0271c996-234d-4291-aa7a-77d43569c247\") " pod="openshift-ingress/kube-auth-proxy-78559bccc4-9vqg5" Oct 30 21:44:58.907451 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:58.907418 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8p59h\" (UniqueName: \"kubernetes.io/projected/0271c996-234d-4291-aa7a-77d43569c247-kube-api-access-8p59h\") pod \"kube-auth-proxy-78559bccc4-9vqg5\" (UID: \"0271c996-234d-4291-aa7a-77d43569c247\") " pod="openshift-ingress/kube-auth-proxy-78559bccc4-9vqg5" Oct 30 21:44:58.998462 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:58.998365 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-x5rxl" event={"ID":"1bb7e9b5-60a0-4cf9-a5ab-2e5e76d077aa","Type":"ContainerStarted","Data":"8ff85a74bc1fdef52b05cdb392720f57eeefa24424a7ab10979e4eaa99a9fdde"} Oct 30 21:44:58.998800 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:58.998668 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-x5rxl" Oct 30 21:44:59.026459 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:59.026435 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:59.039329 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:59.039308 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-78559bccc4-9vqg5" Oct 30 21:44:59.044314 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:59.044262 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-x5rxl" podStartSLOduration=1.0552844239999999 podStartE2EDuration="4.044249454s" totalImagesPullingTime="2.98896503s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:59.042960273 +0000 UTC m=+265.213925458" watchObservedRunningTime="2026-10-30 21:44:59.044249454 +0000 UTC m=+265.215214635" Oct 30 21:44:59.207437 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:59.207407 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:59.319845 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:59.319822 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-78559bccc4-9vqg5"] Oct 30 21:44:59.322217 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:44:59.322194 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0271c996_234d_4291_aa7a_77d43569c247.slice/crio-3dba49bcadea00de8088ad77ba0fb67a5c4aecf1a1c564c59ec8f2cf1dc1a7a9 WatchSource:0}: Error finding container 3dba49bcadea00de8088ad77ba0fb67a5c4aecf1a1c564c59ec8f2cf1dc1a7a9: Status 404 returned error can't find the container with id 3dba49bcadea00de8088ad77ba0fb67a5c4aecf1a1c564c59ec8f2cf1dc1a7a9 Oct 30 21:44:59.355103 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:59.355075 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:44:59.790793 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:44:59.790765 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:00.000700 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:00.000666 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:00.004927 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:00.004893 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-78559bccc4-9vqg5" event={"ID":"0271c996-234d-4291-aa7a-77d43569c247","Type":"ContainerStarted","Data":"3dba49bcadea00de8088ad77ba0fb67a5c4aecf1a1c564c59ec8f2cf1dc1a7a9"} Oct 30 21:45:00.205878 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:00.205839 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:00.511947 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:00.511845 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:00.805863 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:00.805783 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-r5vpw"] Oct 30 21:45:00.812251 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:00.812229 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-r5vpw" Oct 30 21:45:00.819090 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:00.819037 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-r5vpw"] Oct 30 21:45:00.819535 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:00.819516 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-9bf62\"" Oct 30 21:45:00.890611 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:00.890576 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vp5w5\" (UniqueName: \"kubernetes.io/projected/1671230f-01f6-40ef-83a9-9a46f8d0ec7e-kube-api-access-vp5w5\") pod \"authorino-57f7f8659f-r5vpw\" (UID: \"1671230f-01f6-40ef-83a9-9a46f8d0ec7e\") " pod="kuadrant-system/authorino-57f7f8659f-r5vpw" Oct 30 21:45:00.991073 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:00.991043 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp5w5\" (UniqueName: \"kubernetes.io/projected/1671230f-01f6-40ef-83a9-9a46f8d0ec7e-kube-api-access-vp5w5\") pod \"authorino-57f7f8659f-r5vpw\" (UID: \"1671230f-01f6-40ef-83a9-9a46f8d0ec7e\") " pod="kuadrant-system/authorino-57f7f8659f-r5vpw" Oct 30 21:45:00.991539 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:00.991509 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:01.004373 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:01.004350 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp5w5\" (UniqueName: \"kubernetes.io/projected/1671230f-01f6-40ef-83a9-9a46f8d0ec7e-kube-api-access-vp5w5\") pod \"authorino-57f7f8659f-r5vpw\" (UID: \"1671230f-01f6-40ef-83a9-9a46f8d0ec7e\") " pod="kuadrant-system/authorino-57f7f8659f-r5vpw" Oct 30 21:45:01.129418 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:01.129377 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-r5vpw" Oct 30 21:45:01.138248 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:01.138217 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-5cn8v"] Oct 30 21:45:01.142819 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:01.142798 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-5cn8v" Oct 30 21:45:01.170038 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:01.170011 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-5cn8v"] Oct 30 21:45:01.193272 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:01.193070 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzlt6\" (UniqueName: \"kubernetes.io/projected/b250d970-9ecc-4070-8289-46c0ed354b27-kube-api-access-qzlt6\") pod \"authorino-56c6c9ddd5-5cn8v\" (UID: \"b250d970-9ecc-4070-8289-46c0ed354b27\") " pod="kuadrant-system/authorino-56c6c9ddd5-5cn8v" Oct 30 21:45:01.218087 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:01.218051 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:01.293965 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:01.293929 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzlt6\" (UniqueName: \"kubernetes.io/projected/b250d970-9ecc-4070-8289-46c0ed354b27-kube-api-access-qzlt6\") pod \"authorino-56c6c9ddd5-5cn8v\" (UID: \"b250d970-9ecc-4070-8289-46c0ed354b27\") " pod="kuadrant-system/authorino-56c6c9ddd5-5cn8v" Oct 30 21:45:01.306934 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:01.306899 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzlt6\" (UniqueName: \"kubernetes.io/projected/b250d970-9ecc-4070-8289-46c0ed354b27-kube-api-access-qzlt6\") pod \"authorino-56c6c9ddd5-5cn8v\" (UID: \"b250d970-9ecc-4070-8289-46c0ed354b27\") " pod="kuadrant-system/authorino-56c6c9ddd5-5cn8v" Oct 30 21:45:01.414755 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:01.414685 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-r5vpw"] Oct 30 21:45:01.418360 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:45:01.418315 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1671230f_01f6_40ef_83a9_9a46f8d0ec7e.slice/crio-9410594db2b3a02098789d1d6c65490cbf589c55edcb47c8099b01cf5cb8445e WatchSource:0}: Error finding container 9410594db2b3a02098789d1d6c65490cbf589c55edcb47c8099b01cf5cb8445e: Status 404 returned error can't find the container with id 9410594db2b3a02098789d1d6c65490cbf589c55edcb47c8099b01cf5cb8445e Oct 30 21:45:01.471102 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:01.471070 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-5cn8v" Oct 30 21:45:01.618533 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:01.618504 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:01.766103 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:01.766074 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-5cn8v"] Oct 30 21:45:01.767023 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:45:01.766994 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb250d970_9ecc_4070_8289_46c0ed354b27.slice/crio-234a1bd3e1a367c97cbcb3c50b10036b195e5f4380a14391f01f26112966afb6 WatchSource:0}: Error finding container 234a1bd3e1a367c97cbcb3c50b10036b195e5f4380a14391f01f26112966afb6: Status 404 returned error can't find the container with id 234a1bd3e1a367c97cbcb3c50b10036b195e5f4380a14391f01f26112966afb6 Oct 30 21:45:01.812861 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:01.812832 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:01.983622 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:01.983549 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:02.016340 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:02.016311 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-5cn8v" event={"ID":"b250d970-9ecc-4070-8289-46c0ed354b27","Type":"ContainerStarted","Data":"234a1bd3e1a367c97cbcb3c50b10036b195e5f4380a14391f01f26112966afb6"} Oct 30 21:45:02.017705 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:02.017679 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-78559bccc4-9vqg5" event={"ID":"0271c996-234d-4291-aa7a-77d43569c247","Type":"ContainerStarted","Data":"da7ae2e82669890506e254c29a9587e844c2b8abf49b945a5e52cc74a2e88df1"} Oct 30 21:45:02.018800 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:02.018774 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-r5vpw" event={"ID":"1671230f-01f6-40ef-83a9-9a46f8d0ec7e","Type":"ContainerStarted","Data":"9410594db2b3a02098789d1d6c65490cbf589c55edcb47c8099b01cf5cb8445e"} Oct 30 21:45:02.040526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:02.040483 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-78559bccc4-9vqg5" podStartSLOduration=1.5908394559999999 podStartE2EDuration="4.040470383s" totalImagesPullingTime="2.449630927s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:02.03920014 +0000 UTC m=+268.210165321" watchObservedRunningTime="2026-10-30 21:45:02.040470383 +0000 UTC m=+268.211435564" Oct 30 21:45:04.030701 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:04.030598 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-r5vpw" event={"ID":"1671230f-01f6-40ef-83a9-9a46f8d0ec7e","Type":"ContainerStarted","Data":"f100a121ba6e01ced182132e376dfe07b101071801a55b2fc219d301b38b987e"} Oct 30 21:45:04.034778 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:04.034748 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-5cn8v" event={"ID":"b250d970-9ecc-4070-8289-46c0ed354b27","Type":"ContainerStarted","Data":"b7530943d972cafabc11baa9a7b6954e1fd6eac44944bb12936f0339b0ab7ba0"} Oct 30 21:45:04.096891 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:04.096847 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-r5vpw" podStartSLOduration=1.772960775 podStartE2EDuration="4.096833464s" totalImagesPullingTime="2.323872689s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:04.096281863 +0000 UTC m=+270.267247046" watchObservedRunningTime="2026-10-30 21:45:04.096833464 +0000 UTC m=+270.267798624" Oct 30 21:45:04.117017 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:04.116977 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-5cn8v" podStartSLOduration=1.04322917 podStartE2EDuration="3.116964767s" totalImagesPullingTime="2.073735597s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:04.116133019 +0000 UTC m=+270.287098201" watchObservedRunningTime="2026-10-30 21:45:04.116964767 +0000 UTC m=+270.287929964" Oct 30 21:45:04.167028 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:04.166996 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-r5vpw"] Oct 30 21:45:06.043034 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:06.042977 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-r5vpw" podUID="1671230f-01f6-40ef-83a9-9a46f8d0ec7e" containerName="authorino" containerID="cri-o://f100a121ba6e01ced182132e376dfe07b101071801a55b2fc219d301b38b987e" gracePeriod=30 Oct 30 21:45:06.309237 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:06.309161 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-r5vpw" Oct 30 21:45:06.332116 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:06.332080 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1671230f-01f6-40ef-83a9-9a46f8d0ec7e-kube-api-access-vp5w5\" (UniqueName: \"kubernetes.io/projected/1671230f-01f6-40ef-83a9-9a46f8d0ec7e-kube-api-access-vp5w5\") pod \"1671230f-01f6-40ef-83a9-9a46f8d0ec7e\" (UID: \"1671230f-01f6-40ef-83a9-9a46f8d0ec7e\") " Oct 30 21:45:06.334305 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:06.334272 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1671230f-01f6-40ef-83a9-9a46f8d0ec7e-kube-api-access-vp5w5" pod "1671230f-01f6-40ef-83a9-9a46f8d0ec7e" (UID: "1671230f-01f6-40ef-83a9-9a46f8d0ec7e"). InnerVolumeSpecName "kube-api-access-vp5w5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:06.432940 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:06.432911 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vp5w5\" (UniqueName: \"kubernetes.io/projected/1671230f-01f6-40ef-83a9-9a46f8d0ec7e-kube-api-access-vp5w5\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:06.769861 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:06.769825 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:07.000098 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:07.000063 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:07.048673 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:07.048606 6720 generic.go:356] "Generic (PLEG): container finished" podID="1671230f-01f6-40ef-83a9-9a46f8d0ec7e" containerID="f100a121ba6e01ced182132e376dfe07b101071801a55b2fc219d301b38b987e" exitCode=0 Oct 30 21:45:07.048673 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:07.048650 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-r5vpw" Oct 30 21:45:07.048673 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:07.048666 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-r5vpw" event={"ID":"1671230f-01f6-40ef-83a9-9a46f8d0ec7e","Type":"ContainerDied","Data":"f100a121ba6e01ced182132e376dfe07b101071801a55b2fc219d301b38b987e"} Oct 30 21:45:07.049129 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:07.048698 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-r5vpw" event={"ID":"1671230f-01f6-40ef-83a9-9a46f8d0ec7e","Type":"ContainerDied","Data":"9410594db2b3a02098789d1d6c65490cbf589c55edcb47c8099b01cf5cb8445e"} Oct 30 21:45:07.049129 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:07.048723 6720 scope.go:122] "RemoveContainer" containerID="f100a121ba6e01ced182132e376dfe07b101071801a55b2fc219d301b38b987e" Oct 30 21:45:07.057356 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:07.057340 6720 scope.go:122] "RemoveContainer" containerID="f100a121ba6e01ced182132e376dfe07b101071801a55b2fc219d301b38b987e" Oct 30 21:45:07.057690 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:45:07.057668 6720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f100a121ba6e01ced182132e376dfe07b101071801a55b2fc219d301b38b987e\": container with ID starting with f100a121ba6e01ced182132e376dfe07b101071801a55b2fc219d301b38b987e not found: ID does not exist" containerID="f100a121ba6e01ced182132e376dfe07b101071801a55b2fc219d301b38b987e" Oct 30 21:45:07.057767 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:07.057704 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f100a121ba6e01ced182132e376dfe07b101071801a55b2fc219d301b38b987e"} err="failed to get container status \"f100a121ba6e01ced182132e376dfe07b101071801a55b2fc219d301b38b987e\": rpc error: code = NotFound desc = could not find container \"f100a121ba6e01ced182132e376dfe07b101071801a55b2fc219d301b38b987e\": container with ID starting with f100a121ba6e01ced182132e376dfe07b101071801a55b2fc219d301b38b987e not found: ID does not exist" Oct 30 21:45:07.071486 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:07.071462 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-r5vpw"] Oct 30 21:45:07.075099 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:07.075079 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-r5vpw"] Oct 30 21:45:07.954220 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:07.954180 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1671230f-01f6-40ef-83a9-9a46f8d0ec7e" path="/var/lib/kubelet/pods/1671230f-01f6-40ef-83a9-9a46f8d0ec7e/volumes" Oct 30 21:45:10.007017 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:10.006988 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-x5rxl" Oct 30 21:45:10.402421 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:45:10.402371 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 21:45:13.987580 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:13.987487 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:14.133138 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:14.133105 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:16.593354 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:16.593322 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:16.747396 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:16.747352 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:21.998585 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:21.998558 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:22.142906 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:22.142880 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:22.477206 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:22.477176 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:22.636034 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:22.636005 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:22.803614 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:22.803540 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:25.801505 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:25.801470 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:26.864157 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:26.864126 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:27.052640 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:27.052608 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:33.925143 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:33.925116 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:34.074491 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:34.074463 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:39.449708 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:39.449672 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:39.561681 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:39.561640 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-d88xf"] Oct 30 21:45:39.562012 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:39.561990 6720 cpu_manager.go:414] "containerMap: removing container" podUID="1671230f-01f6-40ef-83a9-9a46f8d0ec7e" containerName="authorino" Oct 30 21:45:39.562012 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:39.562004 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1671230f-01f6-40ef-83a9-9a46f8d0ec7e" containerName="authorino" Oct 30 21:45:39.562178 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:39.562074 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="1671230f-01f6-40ef-83a9-9a46f8d0ec7e" containerName="authorino" Oct 30 21:45:39.567681 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:39.567661 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-d88xf" Oct 30 21:45:39.576290 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:39.576263 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-d88xf"] Oct 30 21:45:39.576433 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:39.576416 6720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 30 21:45:39.576484 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:39.576442 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-np89f\"" Oct 30 21:45:39.607301 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:39.607274 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:39.653533 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:39.653506 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-d88xf"] Oct 30 21:45:39.714076 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:39.714007 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c5ba1c23-0ff9-4db5-8a26-d196abd8bdd3-config-file\") pod \"limitador-limitador-74dfcfb894-d88xf\" (UID: \"c5ba1c23-0ff9-4db5-8a26-d196abd8bdd3\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-d88xf" Oct 30 21:45:39.714076 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:39.714051 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jpfz\" (UniqueName: \"kubernetes.io/projected/c5ba1c23-0ff9-4db5-8a26-d196abd8bdd3-kube-api-access-8jpfz\") pod \"limitador-limitador-74dfcfb894-d88xf\" (UID: \"c5ba1c23-0ff9-4db5-8a26-d196abd8bdd3\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-d88xf" Oct 30 21:45:39.780836 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:39.780806 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:39.815022 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:39.814986 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c5ba1c23-0ff9-4db5-8a26-d196abd8bdd3-config-file\") pod \"limitador-limitador-74dfcfb894-d88xf\" (UID: \"c5ba1c23-0ff9-4db5-8a26-d196abd8bdd3\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-d88xf" Oct 30 21:45:39.815185 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:39.815038 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jpfz\" (UniqueName: \"kubernetes.io/projected/c5ba1c23-0ff9-4db5-8a26-d196abd8bdd3-kube-api-access-8jpfz\") pod \"limitador-limitador-74dfcfb894-d88xf\" (UID: \"c5ba1c23-0ff9-4db5-8a26-d196abd8bdd3\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-d88xf" Oct 30 21:45:39.815603 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:39.815584 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c5ba1c23-0ff9-4db5-8a26-d196abd8bdd3-config-file\") pod \"limitador-limitador-74dfcfb894-d88xf\" (UID: \"c5ba1c23-0ff9-4db5-8a26-d196abd8bdd3\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-d88xf" Oct 30 21:45:39.830248 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:39.830226 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jpfz\" (UniqueName: \"kubernetes.io/projected/c5ba1c23-0ff9-4db5-8a26-d196abd8bdd3-kube-api-access-8jpfz\") pod \"limitador-limitador-74dfcfb894-d88xf\" (UID: \"c5ba1c23-0ff9-4db5-8a26-d196abd8bdd3\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-d88xf" Oct 30 21:45:39.880230 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:39.880200 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-d88xf" Oct 30 21:45:40.163395 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:40.163353 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-d88xf"] Oct 30 21:45:40.167467 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:45:40.167441 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5ba1c23_0ff9_4db5_8a26_d196abd8bdd3.slice/crio-8c10ceed560257f12ace981a89e4958a4e9a53ee0738cd63212a3b0e917f176c WatchSource:0}: Error finding container 8c10ceed560257f12ace981a89e4958a4e9a53ee0738cd63212a3b0e917f176c: Status 404 returned error can't find the container with id 8c10ceed560257f12ace981a89e4958a4e9a53ee0738cd63212a3b0e917f176c Oct 30 21:45:40.183104 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:40.183079 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-d88xf" event={"ID":"c5ba1c23-0ff9-4db5-8a26-d196abd8bdd3","Type":"ContainerStarted","Data":"8c10ceed560257f12ace981a89e4958a4e9a53ee0738cd63212a3b0e917f176c"} Oct 30 21:45:40.401153 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:45:40.401113 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 21:45:42.193437 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:42.193399 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-d88xf" event={"ID":"c5ba1c23-0ff9-4db5-8a26-d196abd8bdd3","Type":"ContainerStarted","Data":"0322d8a197f9be6a995f0563892016650f091e4804ef75338a07dfc42e8821eb"} Oct 30 21:45:42.193812 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:42.193502 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-d88xf" Oct 30 21:45:53.198661 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:53.198629 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-d88xf" Oct 30 21:45:53.231442 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:53.231395 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-d88xf" podStartSLOduration=12.697879171 podStartE2EDuration="14.231366743s" totalImagesPullingTime="1.533487572s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:42.268727748 +0000 UTC m=+308.439692930" watchObservedRunningTime="2026-10-30 21:45:53.231366743 +0000 UTC m=+319.402331945" Oct 30 21:45:55.060490 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:55.060454 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-d88xf"] Oct 30 21:45:55.060936 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:55.060646 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-d88xf" podUID="c5ba1c23-0ff9-4db5-8a26-d196abd8bdd3" containerName="limitador" containerID="cri-o://0322d8a197f9be6a995f0563892016650f091e4804ef75338a07dfc42e8821eb" gracePeriod=30 Oct 30 21:45:55.603495 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:55.603473 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-d88xf" Oct 30 21:45:55.757524 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:55.757443 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5ba1c23-0ff9-4db5-8a26-d196abd8bdd3-config-file\" (UniqueName: \"kubernetes.io/configmap/c5ba1c23-0ff9-4db5-8a26-d196abd8bdd3-config-file\") pod \"c5ba1c23-0ff9-4db5-8a26-d196abd8bdd3\" (UID: \"c5ba1c23-0ff9-4db5-8a26-d196abd8bdd3\") " Oct 30 21:45:55.757524 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:55.757497 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5ba1c23-0ff9-4db5-8a26-d196abd8bdd3-kube-api-access-8jpfz\" (UniqueName: \"kubernetes.io/projected/c5ba1c23-0ff9-4db5-8a26-d196abd8bdd3-kube-api-access-8jpfz\") pod \"c5ba1c23-0ff9-4db5-8a26-d196abd8bdd3\" (UID: \"c5ba1c23-0ff9-4db5-8a26-d196abd8bdd3\") " Oct 30 21:45:55.757792 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:55.757769 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5ba1c23-0ff9-4db5-8a26-d196abd8bdd3-config-file" pod "c5ba1c23-0ff9-4db5-8a26-d196abd8bdd3" (UID: "c5ba1c23-0ff9-4db5-8a26-d196abd8bdd3"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:55.759724 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:55.759696 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5ba1c23-0ff9-4db5-8a26-d196abd8bdd3-kube-api-access-8jpfz" pod "c5ba1c23-0ff9-4db5-8a26-d196abd8bdd3" (UID: "c5ba1c23-0ff9-4db5-8a26-d196abd8bdd3"). InnerVolumeSpecName "kube-api-access-8jpfz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:55.858622 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:55.858589 6720 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c5ba1c23-0ff9-4db5-8a26-d196abd8bdd3-config-file\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:55.858622 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:55.858615 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jpfz\" (UniqueName: \"kubernetes.io/projected/c5ba1c23-0ff9-4db5-8a26-d196abd8bdd3-kube-api-access-8jpfz\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:56.124429 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:56.124372 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:56.164280 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:56.164254 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-cptxw"] Oct 30 21:45:56.164651 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:56.164634 6720 cpu_manager.go:414] "containerMap: removing container" podUID="c5ba1c23-0ff9-4db5-8a26-d196abd8bdd3" containerName="limitador" Oct 30 21:45:56.164705 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:56.164654 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5ba1c23-0ff9-4db5-8a26-d196abd8bdd3" containerName="limitador" Oct 30 21:45:56.164768 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:56.164755 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5ba1c23-0ff9-4db5-8a26-d196abd8bdd3" containerName="limitador" Oct 30 21:45:56.167789 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:56.167772 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-cptxw" Oct 30 21:45:56.191006 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:56.190981 6720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 21:45:56.191101 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:56.191051 6720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 21:45:56.199961 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:56.199943 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-k2lk6"] Oct 30 21:45:56.203072 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:56.203055 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-k2lk6" Oct 30 21:45:56.205487 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:56.205469 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-fkz9j\"" Oct 30 21:45:56.213518 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:56.213503 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Oct 30 21:45:56.222837 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:56.222815 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-cptxw"] Oct 30 21:45:56.224341 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:56.224322 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-k2lk6"] Oct 30 21:45:56.255478 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:56.255446 6720 generic.go:356] "Generic (PLEG): container finished" podID="c5ba1c23-0ff9-4db5-8a26-d196abd8bdd3" containerID="0322d8a197f9be6a995f0563892016650f091e4804ef75338a07dfc42e8821eb" exitCode=0 Oct 30 21:45:56.255593 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:56.255493 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-d88xf" event={"ID":"c5ba1c23-0ff9-4db5-8a26-d196abd8bdd3","Type":"ContainerDied","Data":"0322d8a197f9be6a995f0563892016650f091e4804ef75338a07dfc42e8821eb"} Oct 30 21:45:56.255593 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:56.255498 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-d88xf" Oct 30 21:45:56.255593 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:56.255537 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-d88xf" event={"ID":"c5ba1c23-0ff9-4db5-8a26-d196abd8bdd3","Type":"ContainerDied","Data":"8c10ceed560257f12ace981a89e4958a4e9a53ee0738cd63212a3b0e917f176c"} Oct 30 21:45:56.255593 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:56.255560 6720 scope.go:122] "RemoveContainer" containerID="0322d8a197f9be6a995f0563892016650f091e4804ef75338a07dfc42e8821eb" Oct 30 21:45:56.262366 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:56.262345 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9a04fd45-c758-4a59-8c04-c9499862dcbd-cert\") pod \"kserve-controller-manager-776cc66d47-cptxw\" (UID: \"9a04fd45-c758-4a59-8c04-c9499862dcbd\") " pod="opendatahub/kserve-controller-manager-776cc66d47-cptxw" Oct 30 21:45:56.262491 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:56.262403 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmndh\" (UniqueName: \"kubernetes.io/projected/9a04fd45-c758-4a59-8c04-c9499862dcbd-kube-api-access-lmndh\") pod \"kserve-controller-manager-776cc66d47-cptxw\" (UID: \"9a04fd45-c758-4a59-8c04-c9499862dcbd\") " pod="opendatahub/kserve-controller-manager-776cc66d47-cptxw" Oct 30 21:45:56.263759 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:56.263743 6720 scope.go:122] "RemoveContainer" containerID="0322d8a197f9be6a995f0563892016650f091e4804ef75338a07dfc42e8821eb" Oct 30 21:45:56.264018 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:45:56.264001 6720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0322d8a197f9be6a995f0563892016650f091e4804ef75338a07dfc42e8821eb\": container with ID starting with 0322d8a197f9be6a995f0563892016650f091e4804ef75338a07dfc42e8821eb not found: ID does not exist" containerID="0322d8a197f9be6a995f0563892016650f091e4804ef75338a07dfc42e8821eb" Oct 30 21:45:56.264065 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:56.264027 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0322d8a197f9be6a995f0563892016650f091e4804ef75338a07dfc42e8821eb"} err="failed to get container status \"0322d8a197f9be6a995f0563892016650f091e4804ef75338a07dfc42e8821eb\": rpc error: code = NotFound desc = could not find container \"0322d8a197f9be6a995f0563892016650f091e4804ef75338a07dfc42e8821eb\": container with ID starting with 0322d8a197f9be6a995f0563892016650f091e4804ef75338a07dfc42e8821eb not found: ID does not exist" Oct 30 21:45:56.283692 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:56.283669 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-7c7vd"] Oct 30 21:45:56.288328 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:56.288308 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-7c7vd" Oct 30 21:45:56.290025 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:56.290005 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-ckx8d"] Oct 30 21:45:56.293190 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:56.293174 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-ckx8d" Oct 30 21:45:56.307835 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:56.307816 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-7c7vd"] Oct 30 21:45:56.315814 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:56.313432 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-8x86h\"" Oct 30 21:45:56.315814 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:56.313545 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Oct 30 21:45:56.315814 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:56.313601 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Oct 30 21:45:56.315814 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:56.313611 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-7hx4w\"" Oct 30 21:45:56.315814 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:56.313660 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-r8mcn\"" Oct 30 21:45:56.315814 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:56.313775 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-d88xf"] Oct 30 21:45:56.315814 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:56.313790 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Oct 30 21:45:56.317349 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:56.317049 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Oct 30 21:45:56.318412 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:56.318373 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-d88xf"] Oct 30 21:45:56.325486 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:56.325462 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-ckx8d"] Oct 30 21:45:56.363373 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:56.363350 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kskhr\" (UniqueName: \"kubernetes.io/projected/7c9a9adc-e96d-4b46-9399-04a8cdb7eb1c-kube-api-access-kskhr\") pod \"llmisvc-controller-manager-869d5dcc5d-k2lk6\" (UID: \"7c9a9adc-e96d-4b46-9399-04a8cdb7eb1c\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-k2lk6" Oct 30 21:45:56.363480 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:56.363409 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9a04fd45-c758-4a59-8c04-c9499862dcbd-cert\") pod \"kserve-controller-manager-776cc66d47-cptxw\" (UID: \"9a04fd45-c758-4a59-8c04-c9499862dcbd\") " pod="opendatahub/kserve-controller-manager-776cc66d47-cptxw" Oct 30 21:45:56.363480 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:56.363467 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmndh\" (UniqueName: \"kubernetes.io/projected/9a04fd45-c758-4a59-8c04-c9499862dcbd-kube-api-access-lmndh\") pod \"kserve-controller-manager-776cc66d47-cptxw\" (UID: \"9a04fd45-c758-4a59-8c04-c9499862dcbd\") " pod="opendatahub/kserve-controller-manager-776cc66d47-cptxw" Oct 30 21:45:56.363560 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:56.363543 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7c9a9adc-e96d-4b46-9399-04a8cdb7eb1c-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-k2lk6\" (UID: \"7c9a9adc-e96d-4b46-9399-04a8cdb7eb1c\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-k2lk6" Oct 30 21:45:56.365928 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:56.365909 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9a04fd45-c758-4a59-8c04-c9499862dcbd-cert\") pod \"kserve-controller-manager-776cc66d47-cptxw\" (UID: \"9a04fd45-c758-4a59-8c04-c9499862dcbd\") " pod="opendatahub/kserve-controller-manager-776cc66d47-cptxw" Oct 30 21:45:56.380645 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:56.380597 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmndh\" (UniqueName: \"kubernetes.io/projected/9a04fd45-c758-4a59-8c04-c9499862dcbd-kube-api-access-lmndh\") pod \"kserve-controller-manager-776cc66d47-cptxw\" (UID: \"9a04fd45-c758-4a59-8c04-c9499862dcbd\") " pod="opendatahub/kserve-controller-manager-776cc66d47-cptxw" Oct 30 21:45:56.464404 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:56.464354 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd54g\" (UniqueName: \"kubernetes.io/projected/838cabb6-566e-4376-8c49-f0801ac171fc-kube-api-access-kd54g\") pod \"odh-model-controller-769875db56-ckx8d\" (UID: \"838cabb6-566e-4376-8c49-f0801ac171fc\") " pod="opendatahub/odh-model-controller-769875db56-ckx8d" Oct 30 21:45:56.464531 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:56.464414 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kskhr\" (UniqueName: \"kubernetes.io/projected/7c9a9adc-e96d-4b46-9399-04a8cdb7eb1c-kube-api-access-kskhr\") pod \"llmisvc-controller-manager-869d5dcc5d-k2lk6\" (UID: \"7c9a9adc-e96d-4b46-9399-04a8cdb7eb1c\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-k2lk6" Oct 30 21:45:56.464531 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:56.464491 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/838cabb6-566e-4376-8c49-f0801ac171fc-cert\") pod \"odh-model-controller-769875db56-ckx8d\" (UID: \"838cabb6-566e-4376-8c49-f0801ac171fc\") " pod="opendatahub/odh-model-controller-769875db56-ckx8d" Oct 30 21:45:56.464602 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:56.464529 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/838cabb6-566e-4376-8c49-f0801ac171fc-metrics-certs\") pod \"odh-model-controller-769875db56-ckx8d\" (UID: \"838cabb6-566e-4376-8c49-f0801ac171fc\") " pod="opendatahub/odh-model-controller-769875db56-ckx8d" Oct 30 21:45:56.464602 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:56.464578 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7c9a9adc-e96d-4b46-9399-04a8cdb7eb1c-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-k2lk6\" (UID: \"7c9a9adc-e96d-4b46-9399-04a8cdb7eb1c\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-k2lk6" Oct 30 21:45:56.464693 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:56.464665 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cefd95b-2c8c-4271-93b3-4d37e6e00d8e-tls-certs\") pod \"model-serving-api-75bd89fdfc-7c7vd\" (UID: \"0cefd95b-2c8c-4271-93b3-4d37e6e00d8e\") " pod="opendatahub/model-serving-api-75bd89fdfc-7c7vd" Oct 30 21:45:56.464738 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:56.464722 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsk8w\" (UniqueName: \"kubernetes.io/projected/0cefd95b-2c8c-4271-93b3-4d37e6e00d8e-kube-api-access-zsk8w\") pod \"model-serving-api-75bd89fdfc-7c7vd\" (UID: \"0cefd95b-2c8c-4271-93b3-4d37e6e00d8e\") " pod="opendatahub/model-serving-api-75bd89fdfc-7c7vd" Oct 30 21:45:56.467055 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:56.467033 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7c9a9adc-e96d-4b46-9399-04a8cdb7eb1c-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-k2lk6\" (UID: \"7c9a9adc-e96d-4b46-9399-04a8cdb7eb1c\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-k2lk6" Oct 30 21:45:56.480063 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:56.480043 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-cptxw" Oct 30 21:45:56.491091 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:56.491023 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:56.500206 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:56.500184 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kskhr\" (UniqueName: \"kubernetes.io/projected/7c9a9adc-e96d-4b46-9399-04a8cdb7eb1c-kube-api-access-kskhr\") pod \"llmisvc-controller-manager-869d5dcc5d-k2lk6\" (UID: \"7c9a9adc-e96d-4b46-9399-04a8cdb7eb1c\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-k2lk6" Oct 30 21:45:56.515211 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:56.515182 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-k2lk6" Oct 30 21:45:56.565723 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:56.565689 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/838cabb6-566e-4376-8c49-f0801ac171fc-cert\") pod \"odh-model-controller-769875db56-ckx8d\" (UID: \"838cabb6-566e-4376-8c49-f0801ac171fc\") " pod="opendatahub/odh-model-controller-769875db56-ckx8d" Oct 30 21:45:56.565723 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:56.565726 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/838cabb6-566e-4376-8c49-f0801ac171fc-metrics-certs\") pod \"odh-model-controller-769875db56-ckx8d\" (UID: \"838cabb6-566e-4376-8c49-f0801ac171fc\") " pod="opendatahub/odh-model-controller-769875db56-ckx8d" Oct 30 21:45:56.565962 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:56.565760 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cefd95b-2c8c-4271-93b3-4d37e6e00d8e-tls-certs\") pod \"model-serving-api-75bd89fdfc-7c7vd\" (UID: \"0cefd95b-2c8c-4271-93b3-4d37e6e00d8e\") " pod="opendatahub/model-serving-api-75bd89fdfc-7c7vd" Oct 30 21:45:56.565962 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:56.565779 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsk8w\" (UniqueName: \"kubernetes.io/projected/0cefd95b-2c8c-4271-93b3-4d37e6e00d8e-kube-api-access-zsk8w\") pod \"model-serving-api-75bd89fdfc-7c7vd\" (UID: \"0cefd95b-2c8c-4271-93b3-4d37e6e00d8e\") " pod="opendatahub/model-serving-api-75bd89fdfc-7c7vd" Oct 30 21:45:56.565962 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:56.565818 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd54g\" (UniqueName: \"kubernetes.io/projected/838cabb6-566e-4376-8c49-f0801ac171fc-kube-api-access-kd54g\") pod \"odh-model-controller-769875db56-ckx8d\" (UID: \"838cabb6-566e-4376-8c49-f0801ac171fc\") " pod="opendatahub/odh-model-controller-769875db56-ckx8d" Oct 30 21:45:56.568670 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:56.568645 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/838cabb6-566e-4376-8c49-f0801ac171fc-cert\") pod \"odh-model-controller-769875db56-ckx8d\" (UID: \"838cabb6-566e-4376-8c49-f0801ac171fc\") " pod="opendatahub/odh-model-controller-769875db56-ckx8d" Oct 30 21:45:56.568963 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:56.568938 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/838cabb6-566e-4376-8c49-f0801ac171fc-metrics-certs\") pod \"odh-model-controller-769875db56-ckx8d\" (UID: \"838cabb6-566e-4376-8c49-f0801ac171fc\") " pod="opendatahub/odh-model-controller-769875db56-ckx8d" Oct 30 21:45:56.570262 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:56.570241 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cefd95b-2c8c-4271-93b3-4d37e6e00d8e-tls-certs\") pod \"model-serving-api-75bd89fdfc-7c7vd\" (UID: \"0cefd95b-2c8c-4271-93b3-4d37e6e00d8e\") " pod="opendatahub/model-serving-api-75bd89fdfc-7c7vd" Oct 30 21:45:56.592039 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:56.591994 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd54g\" (UniqueName: \"kubernetes.io/projected/838cabb6-566e-4376-8c49-f0801ac171fc-kube-api-access-kd54g\") pod \"odh-model-controller-769875db56-ckx8d\" (UID: \"838cabb6-566e-4376-8c49-f0801ac171fc\") " pod="opendatahub/odh-model-controller-769875db56-ckx8d" Oct 30 21:45:56.597111 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:56.597086 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsk8w\" (UniqueName: \"kubernetes.io/projected/0cefd95b-2c8c-4271-93b3-4d37e6e00d8e-kube-api-access-zsk8w\") pod \"model-serving-api-75bd89fdfc-7c7vd\" (UID: \"0cefd95b-2c8c-4271-93b3-4d37e6e00d8e\") " pod="opendatahub/model-serving-api-75bd89fdfc-7c7vd" Oct 30 21:45:56.601959 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:56.601940 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-7c7vd" Oct 30 21:45:56.608146 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:56.608127 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-ckx8d" Oct 30 21:45:56.764004 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:56.763921 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-cptxw"] Oct 30 21:45:56.766713 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:45:56.766687 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a04fd45_c758_4a59_8c04_c9499862dcbd.slice/crio-0f2020f392bc48b18b0702330b187fbf2dd3927d4c2638f7d39f7155cd79da7b WatchSource:0}: Error finding container 0f2020f392bc48b18b0702330b187fbf2dd3927d4c2638f7d39f7155cd79da7b: Status 404 returned error can't find the container with id 0f2020f392bc48b18b0702330b187fbf2dd3927d4c2638f7d39f7155cd79da7b Oct 30 21:45:56.799439 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:56.799361 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-k2lk6"] Oct 30 21:45:56.801348 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:45:56.801325 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7c9a9adc_e96d_4b46_9399_04a8cdb7eb1c.slice/crio-0ba4549b929f6012408faa970edf5bd5d1ed936cee8b1c146fd1e5187df5ec6a WatchSource:0}: Error finding container 0ba4549b929f6012408faa970edf5bd5d1ed936cee8b1c146fd1e5187df5ec6a: Status 404 returned error can't find the container with id 0ba4549b929f6012408faa970edf5bd5d1ed936cee8b1c146fd1e5187df5ec6a Oct 30 21:45:56.811989 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:56.811969 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:56.882608 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:56.881215 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-7c7vd"] Oct 30 21:45:56.883767 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:45:56.883742 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cefd95b_2c8c_4271_93b3_4d37e6e00d8e.slice/crio-f1a5d1cf72d141392bea2360de93a458b0b5ac3bbe5ab1510817b264d180be50 WatchSource:0}: Error finding container f1a5d1cf72d141392bea2360de93a458b0b5ac3bbe5ab1510817b264d180be50: Status 404 returned error can't find the container with id f1a5d1cf72d141392bea2360de93a458b0b5ac3bbe5ab1510817b264d180be50 Oct 30 21:45:56.902801 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:56.902778 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-ckx8d"] Oct 30 21:45:56.906439 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:45:56.906369 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod838cabb6_566e_4376_8c49_f0801ac171fc.slice/crio-204dd9cdfd6c2cac16e8fa17207f3e1ea41cfee22dfc150e195207d44ca03b2a WatchSource:0}: Error finding container 204dd9cdfd6c2cac16e8fa17207f3e1ea41cfee22dfc150e195207d44ca03b2a: Status 404 returned error can't find the container with id 204dd9cdfd6c2cac16e8fa17207f3e1ea41cfee22dfc150e195207d44ca03b2a Oct 30 21:45:57.263638 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:57.263549 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-k2lk6" event={"ID":"7c9a9adc-e96d-4b46-9399-04a8cdb7eb1c","Type":"ContainerStarted","Data":"0ba4549b929f6012408faa970edf5bd5d1ed936cee8b1c146fd1e5187df5ec6a"} Oct 30 21:45:57.264965 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:57.264934 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-cptxw" event={"ID":"9a04fd45-c758-4a59-8c04-c9499862dcbd","Type":"ContainerStarted","Data":"0f2020f392bc48b18b0702330b187fbf2dd3927d4c2638f7d39f7155cd79da7b"} Oct 30 21:45:57.267800 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:57.267772 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-ckx8d" event={"ID":"838cabb6-566e-4376-8c49-f0801ac171fc","Type":"ContainerStarted","Data":"204dd9cdfd6c2cac16e8fa17207f3e1ea41cfee22dfc150e195207d44ca03b2a"} Oct 30 21:45:57.269195 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:57.269173 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-7c7vd" event={"ID":"0cefd95b-2c8c-4271-93b3-4d37e6e00d8e","Type":"ContainerStarted","Data":"f1a5d1cf72d141392bea2360de93a458b0b5ac3bbe5ab1510817b264d180be50"} Oct 30 21:45:57.576506 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:57.576402 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:57.864721 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:57.864688 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:57.959174 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:57.959142 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5ba1c23-0ff9-4db5-8a26-d196abd8bdd3" path="/var/lib/kubelet/pods/c5ba1c23-0ff9-4db5-8a26-d196abd8bdd3/volumes" Oct 30 21:45:58.115831 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:58.115748 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:58.242051 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:58.242014 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-dgbsj"] Oct 30 21:45:58.245816 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:58.245789 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-dgbsj" Oct 30 21:45:58.255001 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:58.254959 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-mvjs6\"" Oct 30 21:45:58.257504 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:58.257480 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-dgbsj"] Oct 30 21:45:58.265726 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:58.265701 6720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:45:58.266095 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:58.265763 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Oct 30 21:45:58.266095 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:58.265876 6720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:45:58.344653 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:58.344619 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:58.385886 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:58.385629 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/36ab689f-07ab-4d9a-b02b-01a0c2c468d9-data\") pod \"postgres-7cdbf6dc44-dgbsj\" (UID: \"36ab689f-07ab-4d9a-b02b-01a0c2c468d9\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-dgbsj" Oct 30 21:45:58.385886 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:58.385720 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gngqb\" (UniqueName: \"kubernetes.io/projected/36ab689f-07ab-4d9a-b02b-01a0c2c468d9-kube-api-access-gngqb\") pod \"postgres-7cdbf6dc44-dgbsj\" (UID: \"36ab689f-07ab-4d9a-b02b-01a0c2c468d9\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-dgbsj" Oct 30 21:45:58.486800 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:58.486758 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/36ab689f-07ab-4d9a-b02b-01a0c2c468d9-data\") pod \"postgres-7cdbf6dc44-dgbsj\" (UID: \"36ab689f-07ab-4d9a-b02b-01a0c2c468d9\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-dgbsj" Oct 30 21:45:58.486994 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:58.486848 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gngqb\" (UniqueName: \"kubernetes.io/projected/36ab689f-07ab-4d9a-b02b-01a0c2c468d9-kube-api-access-gngqb\") pod \"postgres-7cdbf6dc44-dgbsj\" (UID: \"36ab689f-07ab-4d9a-b02b-01a0c2c468d9\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-dgbsj" Oct 30 21:45:58.487642 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:58.487612 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/36ab689f-07ab-4d9a-b02b-01a0c2c468d9-data\") pod \"postgres-7cdbf6dc44-dgbsj\" (UID: \"36ab689f-07ab-4d9a-b02b-01a0c2c468d9\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-dgbsj" Oct 30 21:45:58.513344 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:58.513275 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gngqb\" (UniqueName: \"kubernetes.io/projected/36ab689f-07ab-4d9a-b02b-01a0c2c468d9-kube-api-access-gngqb\") pod \"postgres-7cdbf6dc44-dgbsj\" (UID: \"36ab689f-07ab-4d9a-b02b-01a0c2c468d9\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-dgbsj" Oct 30 21:45:58.541960 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:58.541929 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:45:58.568584 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:58.568557 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-dgbsj" Oct 30 21:45:58.878554 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:58.878444 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-dgbsj"] Oct 30 21:45:58.890902 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:45:58.890864 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36ab689f_07ab_4d9a_b02b_01a0c2c468d9.slice/crio-e41fae50b562e27d74d909228f5454731185fb87a4af17113cbe099f0d02eed7 WatchSource:0}: Error finding container e41fae50b562e27d74d909228f5454731185fb87a4af17113cbe099f0d02eed7: Status 404 returned error can't find the container with id e41fae50b562e27d74d909228f5454731185fb87a4af17113cbe099f0d02eed7 Oct 30 21:45:59.294228 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:45:59.294158 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-dgbsj" event={"ID":"36ab689f-07ab-4d9a-b02b-01a0c2c468d9","Type":"ContainerStarted","Data":"e41fae50b562e27d74d909228f5454731185fb87a4af17113cbe099f0d02eed7"} Oct 30 21:46:00.309302 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:00.309173 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-k2lk6" event={"ID":"7c9a9adc-e96d-4b46-9399-04a8cdb7eb1c","Type":"ContainerStarted","Data":"80f8b4eaf55c26fcee7217b9720a3a610e46b04a789f73e4ae614bfa9a9926ec"} Oct 30 21:46:00.309302 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:00.309262 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-k2lk6" Oct 30 21:46:00.313740 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:00.313707 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-cptxw" event={"ID":"9a04fd45-c758-4a59-8c04-c9499862dcbd","Type":"ContainerStarted","Data":"ef424d10cdd0a40e847d942884f31610ca9e43d528a6dece2fd7533180418388"} Oct 30 21:46:00.314229 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:00.314205 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-cptxw" Oct 30 21:46:00.316246 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:00.316221 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-7c7vd" event={"ID":"0cefd95b-2c8c-4271-93b3-4d37e6e00d8e","Type":"ContainerStarted","Data":"87e4071979b12afb63f3492e80189dab8719fb262cc22ea1213b15d0c2b27664"} Oct 30 21:46:00.316526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:00.316475 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-7c7vd" Oct 30 21:46:00.354295 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:00.352943 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-k2lk6" podStartSLOduration=1.318068562 podStartE2EDuration="4.352925803s" totalImagesPullingTime="3.034857241s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:00.35102061 +0000 UTC m=+326.521985809" watchObservedRunningTime="2026-10-30 21:46:00.352925803 +0000 UTC m=+326.523890987" Oct 30 21:46:00.374426 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:00.373878 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-7c7vd" podStartSLOduration=1.621225494 podStartE2EDuration="4.373864954s" totalImagesPullingTime="2.75263946s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:00.372825212 +0000 UTC m=+326.543790394" watchObservedRunningTime="2026-10-30 21:46:00.373864954 +0000 UTC m=+326.544830130" Oct 30 21:46:00.396565 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:00.396496 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-cptxw" podStartSLOduration=1.205268549 podStartE2EDuration="4.396479994s" totalImagesPullingTime="3.191211445s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:00.394862213 +0000 UTC m=+326.565827396" watchObservedRunningTime="2026-10-30 21:46:00.396479994 +0000 UTC m=+326.567445177" Oct 30 21:46:01.323056 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:01.323005 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-ckx8d" event={"ID":"838cabb6-566e-4376-8c49-f0801ac171fc","Type":"ContainerStarted","Data":"5b14a4304a9e96d9ef2e999a5fcc2db5932579ea0a2d00e27d3213c7b7ec6685"} Oct 30 21:46:01.323524 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:01.323229 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-ckx8d" Oct 30 21:46:01.343913 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:01.343831 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-ckx8d" podStartSLOduration=1.1577404470000001 podStartE2EDuration="5.343817386s" totalImagesPullingTime="4.186076939s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:01.342994096 +0000 UTC m=+327.513959278" watchObservedRunningTime="2026-10-30 21:46:01.343817386 +0000 UTC m=+327.514782578" Oct 30 21:46:05.346073 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:05.346038 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-dgbsj" event={"ID":"36ab689f-07ab-4d9a-b02b-01a0c2c468d9","Type":"ContainerStarted","Data":"a2ab2a2898910c191ff5abc57b77260fe726a4a617d211efbd96d7821e2a9cec"} Oct 30 21:46:05.346578 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:05.346172 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-dgbsj" Oct 30 21:46:05.365090 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:05.365039 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-dgbsj" podStartSLOduration=1.478472536 podStartE2EDuration="7.365022989s" totalImagesPullingTime="5.886550453s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:05.364015214 +0000 UTC m=+331.534980398" watchObservedRunningTime="2026-10-30 21:46:05.365022989 +0000 UTC m=+331.535988172" Oct 30 21:46:10.077625 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:10.077594 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:10.401772 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:46:10.401736 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 21:46:11.327797 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:11.327768 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-7c7vd" Oct 30 21:46:11.382819 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:11.382789 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-dgbsj" Oct 30 21:46:12.331884 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:12.331854 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-ckx8d" Oct 30 21:46:18.599228 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:18.599192 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:18.609352 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:18.609326 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7b765fc7b7-vj22v"] Oct 30 21:46:18.750551 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:18.750521 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:18.785441 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:18.785419 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7b765fc7b7-vj22v"] Oct 30 21:46:18.785580 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:18.785515 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b765fc7b7-vj22v" Oct 30 21:46:18.791866 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:18.791819 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 21:46:18.792029 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:18.791878 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 21:46:18.792029 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:18.791918 6720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 21:46:18.792029 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:18.791964 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-99v2d\"" Oct 30 21:46:18.872377 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:18.872352 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2083df4e-d61c-4bbc-8089-c9a9f6b5794c-metrics-certs\") pod \"maas-controller-7b765fc7b7-vj22v\" (UID: \"2083df4e-d61c-4bbc-8089-c9a9f6b5794c\") " pod="opendatahub/maas-controller-7b765fc7b7-vj22v" Oct 30 21:46:18.872535 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:18.872408 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfqkp\" (UniqueName: \"kubernetes.io/projected/2083df4e-d61c-4bbc-8089-c9a9f6b5794c-kube-api-access-tfqkp\") pod \"maas-controller-7b765fc7b7-vj22v\" (UID: \"2083df4e-d61c-4bbc-8089-c9a9f6b5794c\") " pod="opendatahub/maas-controller-7b765fc7b7-vj22v" Oct 30 21:46:18.872535 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:18.872524 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2083df4e-d61c-4bbc-8089-c9a9f6b5794c-webhook-cert\") pod \"maas-controller-7b765fc7b7-vj22v\" (UID: \"2083df4e-d61c-4bbc-8089-c9a9f6b5794c\") " pod="opendatahub/maas-controller-7b765fc7b7-vj22v" Oct 30 21:46:18.918039 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:18.918011 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:18.973540 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:18.973507 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2083df4e-d61c-4bbc-8089-c9a9f6b5794c-metrics-certs\") pod \"maas-controller-7b765fc7b7-vj22v\" (UID: \"2083df4e-d61c-4bbc-8089-c9a9f6b5794c\") " pod="opendatahub/maas-controller-7b765fc7b7-vj22v" Oct 30 21:46:18.973690 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:18.973547 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfqkp\" (UniqueName: \"kubernetes.io/projected/2083df4e-d61c-4bbc-8089-c9a9f6b5794c-kube-api-access-tfqkp\") pod \"maas-controller-7b765fc7b7-vj22v\" (UID: \"2083df4e-d61c-4bbc-8089-c9a9f6b5794c\") " pod="opendatahub/maas-controller-7b765fc7b7-vj22v" Oct 30 21:46:18.973690 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:18.973591 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2083df4e-d61c-4bbc-8089-c9a9f6b5794c-webhook-cert\") pod \"maas-controller-7b765fc7b7-vj22v\" (UID: \"2083df4e-d61c-4bbc-8089-c9a9f6b5794c\") " pod="opendatahub/maas-controller-7b765fc7b7-vj22v" Oct 30 21:46:18.976139 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:18.976116 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2083df4e-d61c-4bbc-8089-c9a9f6b5794c-metrics-certs\") pod \"maas-controller-7b765fc7b7-vj22v\" (UID: \"2083df4e-d61c-4bbc-8089-c9a9f6b5794c\") " pod="opendatahub/maas-controller-7b765fc7b7-vj22v" Oct 30 21:46:18.976263 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:18.976167 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2083df4e-d61c-4bbc-8089-c9a9f6b5794c-webhook-cert\") pod \"maas-controller-7b765fc7b7-vj22v\" (UID: \"2083df4e-d61c-4bbc-8089-c9a9f6b5794c\") " pod="opendatahub/maas-controller-7b765fc7b7-vj22v" Oct 30 21:46:18.986251 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:18.986229 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfqkp\" (UniqueName: \"kubernetes.io/projected/2083df4e-d61c-4bbc-8089-c9a9f6b5794c-kube-api-access-tfqkp\") pod \"maas-controller-7b765fc7b7-vj22v\" (UID: \"2083df4e-d61c-4bbc-8089-c9a9f6b5794c\") " pod="opendatahub/maas-controller-7b765fc7b7-vj22v" Oct 30 21:46:19.110557 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:19.110524 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b765fc7b7-vj22v" Oct 30 21:46:19.143197 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:19.143169 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-57d6c9d545-ntxm7"] Oct 30 21:46:19.160363 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:19.160336 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-57d6c9d545-ntxm7"] Oct 30 21:46:19.160504 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:19.160485 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57d6c9d545-ntxm7" Oct 30 21:46:19.276654 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:19.276617 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvwjp\" (UniqueName: \"kubernetes.io/projected/91b3aaaf-cf84-44d0-9156-7174e41f8bed-kube-api-access-wvwjp\") pod \"maas-controller-57d6c9d545-ntxm7\" (UID: \"91b3aaaf-cf84-44d0-9156-7174e41f8bed\") " pod="opendatahub/maas-controller-57d6c9d545-ntxm7" Oct 30 21:46:19.276801 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:19.276673 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91b3aaaf-cf84-44d0-9156-7174e41f8bed-metrics-certs\") pod \"maas-controller-57d6c9d545-ntxm7\" (UID: \"91b3aaaf-cf84-44d0-9156-7174e41f8bed\") " pod="opendatahub/maas-controller-57d6c9d545-ntxm7" Oct 30 21:46:19.276844 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:19.276802 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/91b3aaaf-cf84-44d0-9156-7174e41f8bed-webhook-cert\") pod \"maas-controller-57d6c9d545-ntxm7\" (UID: \"91b3aaaf-cf84-44d0-9156-7174e41f8bed\") " pod="opendatahub/maas-controller-57d6c9d545-ntxm7" Oct 30 21:46:19.378146 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:19.378111 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvwjp\" (UniqueName: \"kubernetes.io/projected/91b3aaaf-cf84-44d0-9156-7174e41f8bed-kube-api-access-wvwjp\") pod \"maas-controller-57d6c9d545-ntxm7\" (UID: \"91b3aaaf-cf84-44d0-9156-7174e41f8bed\") " pod="opendatahub/maas-controller-57d6c9d545-ntxm7" Oct 30 21:46:19.378331 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:19.378158 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91b3aaaf-cf84-44d0-9156-7174e41f8bed-metrics-certs\") pod \"maas-controller-57d6c9d545-ntxm7\" (UID: \"91b3aaaf-cf84-44d0-9156-7174e41f8bed\") " pod="opendatahub/maas-controller-57d6c9d545-ntxm7" Oct 30 21:46:19.378331 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:19.378194 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/91b3aaaf-cf84-44d0-9156-7174e41f8bed-webhook-cert\") pod \"maas-controller-57d6c9d545-ntxm7\" (UID: \"91b3aaaf-cf84-44d0-9156-7174e41f8bed\") " pod="opendatahub/maas-controller-57d6c9d545-ntxm7" Oct 30 21:46:19.380789 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:19.380762 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/91b3aaaf-cf84-44d0-9156-7174e41f8bed-webhook-cert\") pod \"maas-controller-57d6c9d545-ntxm7\" (UID: \"91b3aaaf-cf84-44d0-9156-7174e41f8bed\") " pod="opendatahub/maas-controller-57d6c9d545-ntxm7" Oct 30 21:46:19.380789 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:19.380777 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91b3aaaf-cf84-44d0-9156-7174e41f8bed-metrics-certs\") pod \"maas-controller-57d6c9d545-ntxm7\" (UID: \"91b3aaaf-cf84-44d0-9156-7174e41f8bed\") " pod="opendatahub/maas-controller-57d6c9d545-ntxm7" Oct 30 21:46:19.391120 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:19.391070 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvwjp\" (UniqueName: \"kubernetes.io/projected/91b3aaaf-cf84-44d0-9156-7174e41f8bed-kube-api-access-wvwjp\") pod \"maas-controller-57d6c9d545-ntxm7\" (UID: \"91b3aaaf-cf84-44d0-9156-7174e41f8bed\") " pod="opendatahub/maas-controller-57d6c9d545-ntxm7" Oct 30 21:46:19.417070 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:19.417048 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7b765fc7b7-vj22v"] Oct 30 21:46:19.418782 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:46:19.418755 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2083df4e_d61c_4bbc_8089_c9a9f6b5794c.slice/crio-2066f55160ad271d3f5341388ec75cae355d69726a6e0640ff3d14db498ebeab WatchSource:0}: Error finding container 2066f55160ad271d3f5341388ec75cae355d69726a6e0640ff3d14db498ebeab: Status 404 returned error can't find the container with id 2066f55160ad271d3f5341388ec75cae355d69726a6e0640ff3d14db498ebeab Oct 30 21:46:19.483509 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:19.483486 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57d6c9d545-ntxm7" Oct 30 21:46:19.776352 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:19.776285 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-57d6c9d545-ntxm7"] Oct 30 21:46:19.782058 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:46:19.782020 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91b3aaaf_cf84_44d0_9156_7174e41f8bed.slice/crio-9a4a2ebe1d048494da8c5a310b01ebabbfc412fe45456631f64cb46f7c773a22 WatchSource:0}: Error finding container 9a4a2ebe1d048494da8c5a310b01ebabbfc412fe45456631f64cb46f7c773a22: Status 404 returned error can't find the container with id 9a4a2ebe1d048494da8c5a310b01ebabbfc412fe45456631f64cb46f7c773a22 Oct 30 21:46:20.419324 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:20.419289 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57d6c9d545-ntxm7" event={"ID":"91b3aaaf-cf84-44d0-9156-7174e41f8bed","Type":"ContainerStarted","Data":"9a4a2ebe1d048494da8c5a310b01ebabbfc412fe45456631f64cb46f7c773a22"} Oct 30 21:46:20.420762 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:20.420735 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b765fc7b7-vj22v" event={"ID":"2083df4e-d61c-4bbc-8089-c9a9f6b5794c","Type":"ContainerStarted","Data":"2066f55160ad271d3f5341388ec75cae355d69726a6e0640ff3d14db498ebeab"} Oct 30 21:46:23.096222 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:23.096137 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:23.250914 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:23.250879 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:23.438863 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:23.438825 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b765fc7b7-vj22v" event={"ID":"2083df4e-d61c-4bbc-8089-c9a9f6b5794c","Type":"ContainerStarted","Data":"18a6e79d3deaf4b94f5c7aef9cf4b9a606a70fdabe9aded1780f55a29c2876c1"} Oct 30 21:46:23.439056 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:23.438888 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7b765fc7b7-vj22v" Oct 30 21:46:23.440474 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:23.440447 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57d6c9d545-ntxm7" event={"ID":"91b3aaaf-cf84-44d0-9156-7174e41f8bed","Type":"ContainerStarted","Data":"9a762dd688de8f787adbf5983a25a0f2c1ee5c335777b7dfc191b3bd2fd6d557"} Oct 30 21:46:23.440575 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:23.440558 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-57d6c9d545-ntxm7" Oct 30 21:46:23.501177 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:23.501127 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-7b765fc7b7-vj22v" podStartSLOduration=2.075672247 podStartE2EDuration="5.501113604s" totalImagesPullingTime="3.425441357s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:23.500370048 +0000 UTC m=+349.671335230" watchObservedRunningTime="2026-10-30 21:46:23.501113604 +0000 UTC m=+349.672078780" Oct 30 21:46:23.538618 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:23.538566 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-57d6c9d545-ntxm7" podStartSLOduration=1.484778125 podStartE2EDuration="4.538552514s" totalImagesPullingTime="3.053774389s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:23.537078637 +0000 UTC m=+349.708043819" watchObservedRunningTime="2026-10-30 21:46:23.538552514 +0000 UTC m=+349.709517696" Oct 30 21:46:24.747247 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:24.747216 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:24.920358 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:24.920330 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:25.837542 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:25.837506 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:31.325416 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:31.325372 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-k2lk6" Oct 30 21:46:32.333943 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:32.333913 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-cptxw" Oct 30 21:46:34.446741 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:34.446705 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-57d6c9d545-ntxm7" Oct 30 21:46:34.447140 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:34.446783 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7b765fc7b7-vj22v" Oct 30 21:46:34.526043 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:34.526014 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7b765fc7b7-vj22v"] Oct 30 21:46:34.526314 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:34.526287 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-7b765fc7b7-vj22v" podUID="2083df4e-d61c-4bbc-8089-c9a9f6b5794c" containerName="manager" containerID="cri-o://18a6e79d3deaf4b94f5c7aef9cf4b9a606a70fdabe9aded1780f55a29c2876c1" gracePeriod=30 Oct 30 21:46:34.769162 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:34.769138 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b765fc7b7-vj22v" Oct 30 21:46:34.931537 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:34.931504 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2083df4e-d61c-4bbc-8089-c9a9f6b5794c-kube-api-access-tfqkp\" (UniqueName: \"kubernetes.io/projected/2083df4e-d61c-4bbc-8089-c9a9f6b5794c-kube-api-access-tfqkp\") pod \"2083df4e-d61c-4bbc-8089-c9a9f6b5794c\" (UID: \"2083df4e-d61c-4bbc-8089-c9a9f6b5794c\") " Oct 30 21:46:34.931696 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:34.931575 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2083df4e-d61c-4bbc-8089-c9a9f6b5794c-webhook-cert\" (UniqueName: \"kubernetes.io/secret/2083df4e-d61c-4bbc-8089-c9a9f6b5794c-webhook-cert\") pod \"2083df4e-d61c-4bbc-8089-c9a9f6b5794c\" (UID: \"2083df4e-d61c-4bbc-8089-c9a9f6b5794c\") " Oct 30 21:46:34.931696 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:34.931620 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2083df4e-d61c-4bbc-8089-c9a9f6b5794c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2083df4e-d61c-4bbc-8089-c9a9f6b5794c-metrics-certs\") pod \"2083df4e-d61c-4bbc-8089-c9a9f6b5794c\" (UID: \"2083df4e-d61c-4bbc-8089-c9a9f6b5794c\") " Oct 30 21:46:34.933892 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:34.933855 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2083df4e-d61c-4bbc-8089-c9a9f6b5794c-webhook-cert" pod "2083df4e-d61c-4bbc-8089-c9a9f6b5794c" (UID: "2083df4e-d61c-4bbc-8089-c9a9f6b5794c"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:46:34.934018 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:34.933893 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2083df4e-d61c-4bbc-8089-c9a9f6b5794c-metrics-certs" pod "2083df4e-d61c-4bbc-8089-c9a9f6b5794c" (UID: "2083df4e-d61c-4bbc-8089-c9a9f6b5794c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:46:34.934018 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:34.933936 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2083df4e-d61c-4bbc-8089-c9a9f6b5794c-kube-api-access-tfqkp" pod "2083df4e-d61c-4bbc-8089-c9a9f6b5794c" (UID: "2083df4e-d61c-4bbc-8089-c9a9f6b5794c"). InnerVolumeSpecName "kube-api-access-tfqkp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:35.032488 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:35.032425 6720 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2083df4e-d61c-4bbc-8089-c9a9f6b5794c-webhook-cert\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:46:35.032488 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:35.032448 6720 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2083df4e-d61c-4bbc-8089-c9a9f6b5794c-metrics-certs\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:46:35.032488 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:35.032458 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfqkp\" (UniqueName: \"kubernetes.io/projected/2083df4e-d61c-4bbc-8089-c9a9f6b5794c-kube-api-access-tfqkp\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:46:35.497686 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:35.497655 6720 generic.go:356] "Generic (PLEG): container finished" podID="2083df4e-d61c-4bbc-8089-c9a9f6b5794c" containerID="18a6e79d3deaf4b94f5c7aef9cf4b9a606a70fdabe9aded1780f55a29c2876c1" exitCode=0 Oct 30 21:46:35.498152 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:35.497714 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b765fc7b7-vj22v" Oct 30 21:46:35.498152 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:35.497738 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b765fc7b7-vj22v" event={"ID":"2083df4e-d61c-4bbc-8089-c9a9f6b5794c","Type":"ContainerDied","Data":"18a6e79d3deaf4b94f5c7aef9cf4b9a606a70fdabe9aded1780f55a29c2876c1"} Oct 30 21:46:35.498152 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:35.497792 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b765fc7b7-vj22v" event={"ID":"2083df4e-d61c-4bbc-8089-c9a9f6b5794c","Type":"ContainerDied","Data":"2066f55160ad271d3f5341388ec75cae355d69726a6e0640ff3d14db498ebeab"} Oct 30 21:46:35.498152 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:35.497822 6720 scope.go:122] "RemoveContainer" containerID="18a6e79d3deaf4b94f5c7aef9cf4b9a606a70fdabe9aded1780f55a29c2876c1" Oct 30 21:46:35.506907 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:35.506884 6720 scope.go:122] "RemoveContainer" containerID="18a6e79d3deaf4b94f5c7aef9cf4b9a606a70fdabe9aded1780f55a29c2876c1" Oct 30 21:46:35.507190 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:46:35.507167 6720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18a6e79d3deaf4b94f5c7aef9cf4b9a606a70fdabe9aded1780f55a29c2876c1\": container with ID starting with 18a6e79d3deaf4b94f5c7aef9cf4b9a606a70fdabe9aded1780f55a29c2876c1 not found: ID does not exist" containerID="18a6e79d3deaf4b94f5c7aef9cf4b9a606a70fdabe9aded1780f55a29c2876c1" Oct 30 21:46:35.507285 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:35.507201 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18a6e79d3deaf4b94f5c7aef9cf4b9a606a70fdabe9aded1780f55a29c2876c1"} err="failed to get container status \"18a6e79d3deaf4b94f5c7aef9cf4b9a606a70fdabe9aded1780f55a29c2876c1\": rpc error: code = NotFound desc = could not find container \"18a6e79d3deaf4b94f5c7aef9cf4b9a606a70fdabe9aded1780f55a29c2876c1\": container with ID starting with 18a6e79d3deaf4b94f5c7aef9cf4b9a606a70fdabe9aded1780f55a29c2876c1 not found: ID does not exist" Oct 30 21:46:35.535243 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:35.535221 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7b765fc7b7-vj22v"] Oct 30 21:46:35.540777 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:35.540754 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7b765fc7b7-vj22v"] Oct 30 21:46:35.955170 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:35.955138 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2083df4e-d61c-4bbc-8089-c9a9f6b5794c" path="/var/lib/kubelet/pods/2083df4e-d61c-4bbc-8089-c9a9f6b5794c/volumes" Oct 30 21:46:40.401354 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:46:40.401319 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 21:46:54.848040 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:54.847970 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:55.118111 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:55.118010 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:56.299922 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:56.299891 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:56.346327 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:56.346298 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5cbf66d55-5wjvj"] Oct 30 21:46:56.346698 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:56.346685 6720 cpu_manager.go:414] "containerMap: removing container" podUID="2083df4e-d61c-4bbc-8089-c9a9f6b5794c" containerName="manager" Oct 30 21:46:56.346756 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:56.346699 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2083df4e-d61c-4bbc-8089-c9a9f6b5794c" containerName="manager" Oct 30 21:46:56.346793 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:56.346778 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="2083df4e-d61c-4bbc-8089-c9a9f6b5794c" containerName="manager" Oct 30 21:46:56.352084 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:56.352057 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5cbf66d55-5wjvj" Oct 30 21:46:56.382305 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:56.382264 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5cbf66d55-5wjvj"] Oct 30 21:46:56.384508 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:56.384487 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 30 21:46:56.384639 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:56.384593 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 21:46:56.384639 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:56.384626 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rm8tr\"" Oct 30 21:46:56.413163 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:56.413141 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0a2517a-f83a-42e1-8078-34eed5777c3b-openshift-service-ca\") pod \"maas-api-5cbf66d55-5wjvj\" (UID: \"d0a2517a-f83a-42e1-8078-34eed5777c3b\") " pod="odh-ai-gateway-infra/maas-api-5cbf66d55-5wjvj" Oct 30 21:46:56.413291 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:56.413173 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0a2517a-f83a-42e1-8078-34eed5777c3b-metrics-certs\") pod \"maas-api-5cbf66d55-5wjvj\" (UID: \"d0a2517a-f83a-42e1-8078-34eed5777c3b\") " pod="odh-ai-gateway-infra/maas-api-5cbf66d55-5wjvj" Oct 30 21:46:56.413291 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:56.413275 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0a2517a-f83a-42e1-8078-34eed5777c3b-maas-api-tls\") pod \"maas-api-5cbf66d55-5wjvj\" (UID: \"d0a2517a-f83a-42e1-8078-34eed5777c3b\") " pod="odh-ai-gateway-infra/maas-api-5cbf66d55-5wjvj" Oct 30 21:46:56.413455 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:56.413436 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvxdk\" (UniqueName: \"kubernetes.io/projected/d0a2517a-f83a-42e1-8078-34eed5777c3b-kube-api-access-vvxdk\") pod \"maas-api-5cbf66d55-5wjvj\" (UID: \"d0a2517a-f83a-42e1-8078-34eed5777c3b\") " pod="odh-ai-gateway-infra/maas-api-5cbf66d55-5wjvj" Oct 30 21:46:56.514029 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:56.513996 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0a2517a-f83a-42e1-8078-34eed5777c3b-openshift-service-ca\") pod \"maas-api-5cbf66d55-5wjvj\" (UID: \"d0a2517a-f83a-42e1-8078-34eed5777c3b\") " pod="odh-ai-gateway-infra/maas-api-5cbf66d55-5wjvj" Oct 30 21:46:56.514266 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:56.514037 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0a2517a-f83a-42e1-8078-34eed5777c3b-metrics-certs\") pod \"maas-api-5cbf66d55-5wjvj\" (UID: \"d0a2517a-f83a-42e1-8078-34eed5777c3b\") " pod="odh-ai-gateway-infra/maas-api-5cbf66d55-5wjvj" Oct 30 21:46:56.514266 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:56.514080 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0a2517a-f83a-42e1-8078-34eed5777c3b-maas-api-tls\") pod \"maas-api-5cbf66d55-5wjvj\" (UID: \"d0a2517a-f83a-42e1-8078-34eed5777c3b\") " pod="odh-ai-gateway-infra/maas-api-5cbf66d55-5wjvj" Oct 30 21:46:56.514266 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:56.514123 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvxdk\" (UniqueName: \"kubernetes.io/projected/d0a2517a-f83a-42e1-8078-34eed5777c3b-kube-api-access-vvxdk\") pod \"maas-api-5cbf66d55-5wjvj\" (UID: \"d0a2517a-f83a-42e1-8078-34eed5777c3b\") " pod="odh-ai-gateway-infra/maas-api-5cbf66d55-5wjvj" Oct 30 21:46:56.515005 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:56.514977 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0a2517a-f83a-42e1-8078-34eed5777c3b-openshift-service-ca\") pod \"maas-api-5cbf66d55-5wjvj\" (UID: \"d0a2517a-f83a-42e1-8078-34eed5777c3b\") " pod="odh-ai-gateway-infra/maas-api-5cbf66d55-5wjvj" Oct 30 21:46:56.516830 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:56.516805 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0a2517a-f83a-42e1-8078-34eed5777c3b-metrics-certs\") pod \"maas-api-5cbf66d55-5wjvj\" (UID: \"d0a2517a-f83a-42e1-8078-34eed5777c3b\") " pod="odh-ai-gateway-infra/maas-api-5cbf66d55-5wjvj" Oct 30 21:46:56.516921 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:56.516842 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0a2517a-f83a-42e1-8078-34eed5777c3b-maas-api-tls\") pod \"maas-api-5cbf66d55-5wjvj\" (UID: \"d0a2517a-f83a-42e1-8078-34eed5777c3b\") " pod="odh-ai-gateway-infra/maas-api-5cbf66d55-5wjvj" Oct 30 21:46:56.528004 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:56.527987 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvxdk\" (UniqueName: \"kubernetes.io/projected/d0a2517a-f83a-42e1-8078-34eed5777c3b-kube-api-access-vvxdk\") pod \"maas-api-5cbf66d55-5wjvj\" (UID: \"d0a2517a-f83a-42e1-8078-34eed5777c3b\") " pod="odh-ai-gateway-infra/maas-api-5cbf66d55-5wjvj" Oct 30 21:46:56.576160 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:56.576104 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:46:56.665667 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:56.665634 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5cbf66d55-5wjvj" Oct 30 21:46:56.951311 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:56.951150 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5cbf66d55-5wjvj"] Oct 30 21:46:56.954107 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:46:56.954076 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0a2517a_f83a_42e1_8078_34eed5777c3b.slice/crio-05c56cb6c834fce6559df02a5d72264b534ae4b2697df1de9db7fe8fdd057b20 WatchSource:0}: Error finding container 05c56cb6c834fce6559df02a5d72264b534ae4b2697df1de9db7fe8fdd057b20: Status 404 returned error can't find the container with id 05c56cb6c834fce6559df02a5d72264b534ae4b2697df1de9db7fe8fdd057b20 Oct 30 21:46:57.597771 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:46:57.597736 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5cbf66d55-5wjvj" event={"ID":"d0a2517a-f83a-42e1-8078-34eed5777c3b","Type":"ContainerStarted","Data":"05c56cb6c834fce6559df02a5d72264b534ae4b2697df1de9db7fe8fdd057b20"} Oct 30 21:47:01.619108 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:01.619076 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5cbf66d55-5wjvj" event={"ID":"d0a2517a-f83a-42e1-8078-34eed5777c3b","Type":"ContainerStarted","Data":"eb3e30486a245ddf9437df1886a4081cb7c2c35f9f93364123e310cd8e122a84"} Oct 30 21:47:01.619522 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:01.619192 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5cbf66d55-5wjvj" Oct 30 21:47:01.644110 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:01.644062 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5cbf66d55-5wjvj" podStartSLOduration=1.942348767 podStartE2EDuration="5.64404952s" totalImagesPullingTime="3.701700753s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:01.642699058 +0000 UTC m=+387.813664239" watchObservedRunningTime="2026-10-30 21:47:01.64404952 +0000 UTC m=+387.815014703" Oct 30 21:47:05.785257 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:05.785225 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:07.629246 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:07.629219 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5cbf66d55-5wjvj" Oct 30 21:47:08.145568 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:08.145544 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:08.337914 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:08.337882 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:08.445851 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:08.445782 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-nhr4g"] Oct 30 21:47:08.449601 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:08.449580 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-nhr4g" Oct 30 21:47:08.459945 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:08.459920 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-nhr4g"] Oct 30 21:47:08.520828 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:08.520798 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptwqg\" (UniqueName: \"kubernetes.io/projected/e682a4ad-36ba-48e0-af8c-eea8b4748c98-kube-api-access-ptwqg\") pod \"authorino-6f6944bcdd-nhr4g\" (UID: \"e682a4ad-36ba-48e0-af8c-eea8b4748c98\") " pod="kuadrant-system/authorino-6f6944bcdd-nhr4g" Oct 30 21:47:08.529698 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:08.529671 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-nhr4g"] Oct 30 21:47:08.529871 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:47:08.529854 6720 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-ptwqg], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6f6944bcdd-nhr4g" podUID="e682a4ad-36ba-48e0-af8c-eea8b4748c98" Oct 30 21:47:08.558632 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:08.558605 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-w658c"] Oct 30 21:47:08.562065 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:08.562047 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-w658c" Oct 30 21:47:08.570957 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:08.570939 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Oct 30 21:47:08.578169 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:08.578148 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-w658c"] Oct 30 21:47:08.621632 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:08.621605 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptwqg\" (UniqueName: \"kubernetes.io/projected/e682a4ad-36ba-48e0-af8c-eea8b4748c98-kube-api-access-ptwqg\") pod \"authorino-6f6944bcdd-nhr4g\" (UID: \"e682a4ad-36ba-48e0-af8c-eea8b4748c98\") " pod="kuadrant-system/authorino-6f6944bcdd-nhr4g" Oct 30 21:47:08.621755 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:08.621643 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/971ae5db-755c-4a5d-8012-22c4d54efab8-tls-cert\") pod \"authorino-6889cbd8c4-w658c\" (UID: \"971ae5db-755c-4a5d-8012-22c4d54efab8\") " pod="kuadrant-system/authorino-6889cbd8c4-w658c" Oct 30 21:47:08.621813 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:08.621796 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp8hw\" (UniqueName: \"kubernetes.io/projected/971ae5db-755c-4a5d-8012-22c4d54efab8-kube-api-access-qp8hw\") pod \"authorino-6889cbd8c4-w658c\" (UID: \"971ae5db-755c-4a5d-8012-22c4d54efab8\") " pod="kuadrant-system/authorino-6889cbd8c4-w658c" Oct 30 21:47:08.639026 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:08.638996 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptwqg\" (UniqueName: \"kubernetes.io/projected/e682a4ad-36ba-48e0-af8c-eea8b4748c98-kube-api-access-ptwqg\") pod \"authorino-6f6944bcdd-nhr4g\" (UID: \"e682a4ad-36ba-48e0-af8c-eea8b4748c98\") " pod="kuadrant-system/authorino-6f6944bcdd-nhr4g" Oct 30 21:47:08.649454 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:08.649434 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-nhr4g" Oct 30 21:47:08.656497 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:08.656480 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-nhr4g" Oct 30 21:47:08.722122 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:08.722052 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e682a4ad-36ba-48e0-af8c-eea8b4748c98-kube-api-access-ptwqg\" (UniqueName: \"kubernetes.io/projected/e682a4ad-36ba-48e0-af8c-eea8b4748c98-kube-api-access-ptwqg\") pod \"e682a4ad-36ba-48e0-af8c-eea8b4748c98\" (UID: \"e682a4ad-36ba-48e0-af8c-eea8b4748c98\") " Oct 30 21:47:08.722285 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:08.722267 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/971ae5db-755c-4a5d-8012-22c4d54efab8-tls-cert\") pod \"authorino-6889cbd8c4-w658c\" (UID: \"971ae5db-755c-4a5d-8012-22c4d54efab8\") " pod="kuadrant-system/authorino-6889cbd8c4-w658c" Oct 30 21:47:08.722377 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:08.722363 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp8hw\" (UniqueName: \"kubernetes.io/projected/971ae5db-755c-4a5d-8012-22c4d54efab8-kube-api-access-qp8hw\") pod \"authorino-6889cbd8c4-w658c\" (UID: \"971ae5db-755c-4a5d-8012-22c4d54efab8\") " pod="kuadrant-system/authorino-6889cbd8c4-w658c" Oct 30 21:47:08.724226 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:08.724199 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e682a4ad-36ba-48e0-af8c-eea8b4748c98-kube-api-access-ptwqg" pod "e682a4ad-36ba-48e0-af8c-eea8b4748c98" (UID: "e682a4ad-36ba-48e0-af8c-eea8b4748c98"). InnerVolumeSpecName "kube-api-access-ptwqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:47:08.724700 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:08.724682 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/971ae5db-755c-4a5d-8012-22c4d54efab8-tls-cert\") pod \"authorino-6889cbd8c4-w658c\" (UID: \"971ae5db-755c-4a5d-8012-22c4d54efab8\") " pod="kuadrant-system/authorino-6889cbd8c4-w658c" Oct 30 21:47:08.735469 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:08.735449 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp8hw\" (UniqueName: \"kubernetes.io/projected/971ae5db-755c-4a5d-8012-22c4d54efab8-kube-api-access-qp8hw\") pod \"authorino-6889cbd8c4-w658c\" (UID: \"971ae5db-755c-4a5d-8012-22c4d54efab8\") " pod="kuadrant-system/authorino-6889cbd8c4-w658c" Oct 30 21:47:08.823735 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:08.823704 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptwqg\" (UniqueName: \"kubernetes.io/projected/e682a4ad-36ba-48e0-af8c-eea8b4748c98-kube-api-access-ptwqg\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:08.868754 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:08.868720 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-w658c"] Oct 30 21:47:08.868958 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:08.868945 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-w658c" Oct 30 21:47:08.914111 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:08.914075 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-f77ccdd9f-2w568"] Oct 30 21:47:08.918581 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:08.918558 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-f77ccdd9f-2w568" Oct 30 21:47:08.930075 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:08.930044 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-f77ccdd9f-2w568"] Oct 30 21:47:09.025961 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:09.025869 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9476a84f-8210-47db-a3f1-66f5a5eea2db-tls-cert\") pod \"authorino-f77ccdd9f-2w568\" (UID: \"9476a84f-8210-47db-a3f1-66f5a5eea2db\") " pod="kuadrant-system/authorino-f77ccdd9f-2w568" Oct 30 21:47:09.026130 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:09.026007 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h826z\" (UniqueName: \"kubernetes.io/projected/9476a84f-8210-47db-a3f1-66f5a5eea2db-kube-api-access-h826z\") pod \"authorino-f77ccdd9f-2w568\" (UID: \"9476a84f-8210-47db-a3f1-66f5a5eea2db\") " pod="kuadrant-system/authorino-f77ccdd9f-2w568" Oct 30 21:47:09.127508 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:09.127467 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h826z\" (UniqueName: \"kubernetes.io/projected/9476a84f-8210-47db-a3f1-66f5a5eea2db-kube-api-access-h826z\") pod \"authorino-f77ccdd9f-2w568\" (UID: \"9476a84f-8210-47db-a3f1-66f5a5eea2db\") " pod="kuadrant-system/authorino-f77ccdd9f-2w568" Oct 30 21:47:09.127683 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:09.127565 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9476a84f-8210-47db-a3f1-66f5a5eea2db-tls-cert\") pod \"authorino-f77ccdd9f-2w568\" (UID: \"9476a84f-8210-47db-a3f1-66f5a5eea2db\") " pod="kuadrant-system/authorino-f77ccdd9f-2w568" Oct 30 21:47:09.130631 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:09.130603 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9476a84f-8210-47db-a3f1-66f5a5eea2db-tls-cert\") pod \"authorino-f77ccdd9f-2w568\" (UID: \"9476a84f-8210-47db-a3f1-66f5a5eea2db\") " pod="kuadrant-system/authorino-f77ccdd9f-2w568" Oct 30 21:47:09.141114 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:09.141086 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h826z\" (UniqueName: \"kubernetes.io/projected/9476a84f-8210-47db-a3f1-66f5a5eea2db-kube-api-access-h826z\") pod \"authorino-f77ccdd9f-2w568\" (UID: \"9476a84f-8210-47db-a3f1-66f5a5eea2db\") " pod="kuadrant-system/authorino-f77ccdd9f-2w568" Oct 30 21:47:09.148069 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:09.148047 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-w658c"] Oct 30 21:47:09.150307 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:47:09.150277 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod971ae5db_755c_4a5d_8012_22c4d54efab8.slice/crio-8b6cff4d239454703271d1af9fca8050ed974f71c7185002a6556b24aa3c2839 WatchSource:0}: Error finding container 8b6cff4d239454703271d1af9fca8050ed974f71c7185002a6556b24aa3c2839: Status 404 returned error can't find the container with id 8b6cff4d239454703271d1af9fca8050ed974f71c7185002a6556b24aa3c2839 Oct 30 21:47:09.241745 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:09.241712 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-f77ccdd9f-2w568" Oct 30 21:47:09.534526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:09.534374 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-f77ccdd9f-2w568"] Oct 30 21:47:09.536674 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:47:09.536648 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9476a84f_8210_47db_a3f1_66f5a5eea2db.slice/crio-d2bb34f2e3bea346f3dcef024b1674bea836cb9147f32a366aad70a747ef52ed WatchSource:0}: Error finding container d2bb34f2e3bea346f3dcef024b1674bea836cb9147f32a366aad70a747ef52ed: Status 404 returned error can't find the container with id d2bb34f2e3bea346f3dcef024b1674bea836cb9147f32a366aad70a747ef52ed Oct 30 21:47:09.654539 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:09.654498 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-f77ccdd9f-2w568" event={"ID":"9476a84f-8210-47db-a3f1-66f5a5eea2db","Type":"ContainerStarted","Data":"d2bb34f2e3bea346f3dcef024b1674bea836cb9147f32a366aad70a747ef52ed"} Oct 30 21:47:09.655706 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:09.655680 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-w658c" event={"ID":"971ae5db-755c-4a5d-8012-22c4d54efab8","Type":"ContainerStarted","Data":"8b6cff4d239454703271d1af9fca8050ed974f71c7185002a6556b24aa3c2839"} Oct 30 21:47:09.655828 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:09.655708 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-nhr4g" Oct 30 21:47:09.691086 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:09.691058 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-nhr4g"] Oct 30 21:47:09.694258 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:09.694236 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-nhr4g"] Oct 30 21:47:09.954965 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:09.954935 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e682a4ad-36ba-48e0-af8c-eea8b4748c98" path="/var/lib/kubelet/pods/e682a4ad-36ba-48e0-af8c-eea8b4748c98/volumes" Oct 30 21:47:10.413079 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:47:10.413036 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 21:47:10.663191 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:10.663083 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-f77ccdd9f-2w568" event={"ID":"9476a84f-8210-47db-a3f1-66f5a5eea2db","Type":"ContainerStarted","Data":"2f4556627b5c0071e9652048eb78e1fa3bb72e31d690dd1a6fe7ed3b4e760398"} Oct 30 21:47:10.665287 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:10.665236 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-w658c" event={"ID":"971ae5db-755c-4a5d-8012-22c4d54efab8","Type":"ContainerStarted","Data":"8129118d7ed00e23fb95b55655f4e254c9f7f3a4ac68567b44a6521ac1242bad"} Oct 30 21:47:10.665446 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:10.665306 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6889cbd8c4-w658c" podUID="971ae5db-755c-4a5d-8012-22c4d54efab8" containerName="authorino" containerID="cri-o://8129118d7ed00e23fb95b55655f4e254c9f7f3a4ac68567b44a6521ac1242bad" gracePeriod=30 Oct 30 21:47:10.686947 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:10.686905 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-f77ccdd9f-2w568" podStartSLOduration=2.10386821 podStartE2EDuration="2.686891542s" totalImagesPullingTime="583.023332ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:10.684915655 +0000 UTC m=+396.855880861" watchObservedRunningTime="2026-10-30 21:47:10.686891542 +0000 UTC m=+396.857856721" Oct 30 21:47:10.732144 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:10.732079 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6889cbd8c4-w658c" podStartSLOduration=1.922392977 podStartE2EDuration="2.732057791s" totalImagesPullingTime="809.664814ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:10.706107122 +0000 UTC m=+396.877072305" watchObservedRunningTime="2026-10-30 21:47:10.732057791 +0000 UTC m=+396.903022975" Oct 30 21:47:10.734187 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:10.734155 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-5cn8v"] Oct 30 21:47:10.734431 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:10.734405 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-5cn8v" podUID="b250d970-9ecc-4070-8289-46c0ed354b27" containerName="authorino" containerID="cri-o://b7530943d972cafabc11baa9a7b6954e1fd6eac44944bb12936f0339b0ab7ba0" gracePeriod=30 Oct 30 21:47:10.983913 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:10.983890 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-w658c" Oct 30 21:47:11.002558 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:11.002538 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-5cn8v" Oct 30 21:47:11.045934 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:11.045907 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/971ae5db-755c-4a5d-8012-22c4d54efab8-kube-api-access-qp8hw\" (UniqueName: \"kubernetes.io/projected/971ae5db-755c-4a5d-8012-22c4d54efab8-kube-api-access-qp8hw\") pod \"971ae5db-755c-4a5d-8012-22c4d54efab8\" (UID: \"971ae5db-755c-4a5d-8012-22c4d54efab8\") " Oct 30 21:47:11.046140 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:11.046051 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b250d970-9ecc-4070-8289-46c0ed354b27-kube-api-access-qzlt6\" (UniqueName: \"kubernetes.io/projected/b250d970-9ecc-4070-8289-46c0ed354b27-kube-api-access-qzlt6\") pod \"b250d970-9ecc-4070-8289-46c0ed354b27\" (UID: \"b250d970-9ecc-4070-8289-46c0ed354b27\") " Oct 30 21:47:11.046140 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:11.046108 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/971ae5db-755c-4a5d-8012-22c4d54efab8-tls-cert\" (UniqueName: \"kubernetes.io/secret/971ae5db-755c-4a5d-8012-22c4d54efab8-tls-cert\") pod \"971ae5db-755c-4a5d-8012-22c4d54efab8\" (UID: \"971ae5db-755c-4a5d-8012-22c4d54efab8\") " Oct 30 21:47:11.048605 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:11.048580 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/971ae5db-755c-4a5d-8012-22c4d54efab8-kube-api-access-qp8hw" pod "971ae5db-755c-4a5d-8012-22c4d54efab8" (UID: "971ae5db-755c-4a5d-8012-22c4d54efab8"). InnerVolumeSpecName "kube-api-access-qp8hw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:47:11.048743 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:11.048721 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b250d970-9ecc-4070-8289-46c0ed354b27-kube-api-access-qzlt6" pod "b250d970-9ecc-4070-8289-46c0ed354b27" (UID: "b250d970-9ecc-4070-8289-46c0ed354b27"). InnerVolumeSpecName "kube-api-access-qzlt6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:47:11.057555 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:11.057530 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/971ae5db-755c-4a5d-8012-22c4d54efab8-tls-cert" pod "971ae5db-755c-4a5d-8012-22c4d54efab8" (UID: "971ae5db-755c-4a5d-8012-22c4d54efab8"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:47:11.147647 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:11.147618 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzlt6\" (UniqueName: \"kubernetes.io/projected/b250d970-9ecc-4070-8289-46c0ed354b27-kube-api-access-qzlt6\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:11.147647 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:11.147645 6720 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/971ae5db-755c-4a5d-8012-22c4d54efab8-tls-cert\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:11.147828 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:11.147655 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qp8hw\" (UniqueName: \"kubernetes.io/projected/971ae5db-755c-4a5d-8012-22c4d54efab8-kube-api-access-qp8hw\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:11.671895 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:11.671860 6720 generic.go:356] "Generic (PLEG): container finished" podID="b250d970-9ecc-4070-8289-46c0ed354b27" containerID="b7530943d972cafabc11baa9a7b6954e1fd6eac44944bb12936f0339b0ab7ba0" exitCode=0 Oct 30 21:47:11.672345 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:11.671907 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-5cn8v" Oct 30 21:47:11.672345 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:11.671946 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-5cn8v" event={"ID":"b250d970-9ecc-4070-8289-46c0ed354b27","Type":"ContainerDied","Data":"b7530943d972cafabc11baa9a7b6954e1fd6eac44944bb12936f0339b0ab7ba0"} Oct 30 21:47:11.672345 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:11.671989 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-5cn8v" event={"ID":"b250d970-9ecc-4070-8289-46c0ed354b27","Type":"ContainerDied","Data":"234a1bd3e1a367c97cbcb3c50b10036b195e5f4380a14391f01f26112966afb6"} Oct 30 21:47:11.672345 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:11.672007 6720 scope.go:122] "RemoveContainer" containerID="b7530943d972cafabc11baa9a7b6954e1fd6eac44944bb12936f0339b0ab7ba0" Oct 30 21:47:11.673345 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:11.673324 6720 generic.go:356] "Generic (PLEG): container finished" podID="971ae5db-755c-4a5d-8012-22c4d54efab8" containerID="8129118d7ed00e23fb95b55655f4e254c9f7f3a4ac68567b44a6521ac1242bad" exitCode=0 Oct 30 21:47:11.673483 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:11.673364 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-w658c" Oct 30 21:47:11.673483 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:11.673376 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-w658c" event={"ID":"971ae5db-755c-4a5d-8012-22c4d54efab8","Type":"ContainerDied","Data":"8129118d7ed00e23fb95b55655f4e254c9f7f3a4ac68567b44a6521ac1242bad"} Oct 30 21:47:11.673483 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:11.673447 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6889cbd8c4-w658c" event={"ID":"971ae5db-755c-4a5d-8012-22c4d54efab8","Type":"ContainerDied","Data":"8b6cff4d239454703271d1af9fca8050ed974f71c7185002a6556b24aa3c2839"} Oct 30 21:47:11.680852 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:11.680821 6720 scope.go:122] "RemoveContainer" containerID="b7530943d972cafabc11baa9a7b6954e1fd6eac44944bb12936f0339b0ab7ba0" Oct 30 21:47:11.681315 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:47:11.681115 6720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7530943d972cafabc11baa9a7b6954e1fd6eac44944bb12936f0339b0ab7ba0\": container with ID starting with b7530943d972cafabc11baa9a7b6954e1fd6eac44944bb12936f0339b0ab7ba0 not found: ID does not exist" containerID="b7530943d972cafabc11baa9a7b6954e1fd6eac44944bb12936f0339b0ab7ba0" Oct 30 21:47:11.681315 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:11.681137 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7530943d972cafabc11baa9a7b6954e1fd6eac44944bb12936f0339b0ab7ba0"} err="failed to get container status \"b7530943d972cafabc11baa9a7b6954e1fd6eac44944bb12936f0339b0ab7ba0\": rpc error: code = NotFound desc = could not find container \"b7530943d972cafabc11baa9a7b6954e1fd6eac44944bb12936f0339b0ab7ba0\": container with ID starting with b7530943d972cafabc11baa9a7b6954e1fd6eac44944bb12936f0339b0ab7ba0 not found: ID does not exist" Oct 30 21:47:11.681315 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:11.681159 6720 scope.go:122] "RemoveContainer" containerID="8129118d7ed00e23fb95b55655f4e254c9f7f3a4ac68567b44a6521ac1242bad" Oct 30 21:47:11.689082 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:11.689066 6720 scope.go:122] "RemoveContainer" containerID="8129118d7ed00e23fb95b55655f4e254c9f7f3a4ac68567b44a6521ac1242bad" Oct 30 21:47:11.689351 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:47:11.689333 6720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8129118d7ed00e23fb95b55655f4e254c9f7f3a4ac68567b44a6521ac1242bad\": container with ID starting with 8129118d7ed00e23fb95b55655f4e254c9f7f3a4ac68567b44a6521ac1242bad not found: ID does not exist" containerID="8129118d7ed00e23fb95b55655f4e254c9f7f3a4ac68567b44a6521ac1242bad" Oct 30 21:47:11.689426 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:11.689360 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8129118d7ed00e23fb95b55655f4e254c9f7f3a4ac68567b44a6521ac1242bad"} err="failed to get container status \"8129118d7ed00e23fb95b55655f4e254c9f7f3a4ac68567b44a6521ac1242bad\": rpc error: code = NotFound desc = could not find container \"8129118d7ed00e23fb95b55655f4e254c9f7f3a4ac68567b44a6521ac1242bad\": container with ID starting with 8129118d7ed00e23fb95b55655f4e254c9f7f3a4ac68567b44a6521ac1242bad not found: ID does not exist" Oct 30 21:47:11.699306 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:11.699289 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-5cn8v"] Oct 30 21:47:11.701029 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:11.701010 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-5cn8v"] Oct 30 21:47:11.716884 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:11.716858 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-w658c"] Oct 30 21:47:11.723220 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:11.723201 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-w658c"] Oct 30 21:47:11.745291 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:11.745271 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-phjxr"] Oct 30 21:47:11.745690 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:11.745678 6720 cpu_manager.go:414] "containerMap: removing container" podUID="b250d970-9ecc-4070-8289-46c0ed354b27" containerName="authorino" Oct 30 21:47:11.745737 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:11.745692 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b250d970-9ecc-4070-8289-46c0ed354b27" containerName="authorino" Oct 30 21:47:11.745737 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:11.745719 6720 cpu_manager.go:414] "containerMap: removing container" podUID="971ae5db-755c-4a5d-8012-22c4d54efab8" containerName="authorino" Oct 30 21:47:11.745737 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:11.745724 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="971ae5db-755c-4a5d-8012-22c4d54efab8" containerName="authorino" Oct 30 21:47:11.745820 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:11.745783 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="b250d970-9ecc-4070-8289-46c0ed354b27" containerName="authorino" Oct 30 21:47:11.745820 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:11.745794 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="971ae5db-755c-4a5d-8012-22c4d54efab8" containerName="authorino" Oct 30 21:47:11.754088 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:11.754065 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-phjxr" Oct 30 21:47:11.762080 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:11.762062 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-phjxr"] Oct 30 21:47:11.854673 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:11.854636 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/753e9e9a-5c68-4540-8fc7-858514ca428e-cert\") pod \"llmisvc-controller-manager-6466c554d8-phjxr\" (UID: \"753e9e9a-5c68-4540-8fc7-858514ca428e\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-phjxr" Oct 30 21:47:11.854673 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:11.854669 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dz496\" (UniqueName: \"kubernetes.io/projected/753e9e9a-5c68-4540-8fc7-858514ca428e-kube-api-access-dz496\") pod \"llmisvc-controller-manager-6466c554d8-phjxr\" (UID: \"753e9e9a-5c68-4540-8fc7-858514ca428e\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-phjxr" Oct 30 21:47:11.955056 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:11.954985 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="971ae5db-755c-4a5d-8012-22c4d54efab8" path="/var/lib/kubelet/pods/971ae5db-755c-4a5d-8012-22c4d54efab8/volumes" Oct 30 21:47:11.955228 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:11.955209 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/753e9e9a-5c68-4540-8fc7-858514ca428e-cert\") pod \"llmisvc-controller-manager-6466c554d8-phjxr\" (UID: \"753e9e9a-5c68-4540-8fc7-858514ca428e\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-phjxr" Oct 30 21:47:11.955270 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:11.955238 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz496\" (UniqueName: \"kubernetes.io/projected/753e9e9a-5c68-4540-8fc7-858514ca428e-kube-api-access-dz496\") pod \"llmisvc-controller-manager-6466c554d8-phjxr\" (UID: \"753e9e9a-5c68-4540-8fc7-858514ca428e\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-phjxr" Oct 30 21:47:11.955305 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:11.955274 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b250d970-9ecc-4070-8289-46c0ed354b27" path="/var/lib/kubelet/pods/b250d970-9ecc-4070-8289-46c0ed354b27/volumes" Oct 30 21:47:11.957745 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:11.957719 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/753e9e9a-5c68-4540-8fc7-858514ca428e-cert\") pod \"llmisvc-controller-manager-6466c554d8-phjxr\" (UID: \"753e9e9a-5c68-4540-8fc7-858514ca428e\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-phjxr" Oct 30 21:47:11.968446 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:11.968423 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz496\" (UniqueName: \"kubernetes.io/projected/753e9e9a-5c68-4540-8fc7-858514ca428e-kube-api-access-dz496\") pod \"llmisvc-controller-manager-6466c554d8-phjxr\" (UID: \"753e9e9a-5c68-4540-8fc7-858514ca428e\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-phjxr" Oct 30 21:47:12.068555 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:12.068528 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-phjxr" Oct 30 21:47:12.344919 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:12.344885 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-phjxr"] Oct 30 21:47:12.347434 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:47:12.347401 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod753e9e9a_5c68_4540_8fc7_858514ca428e.slice/crio-5915fb03d92992b0a79d253956e0c461eb97ee74aa32dd4c02b28fc325b088df WatchSource:0}: Error finding container 5915fb03d92992b0a79d253956e0c461eb97ee74aa32dd4c02b28fc325b088df: Status 404 returned error can't find the container with id 5915fb03d92992b0a79d253956e0c461eb97ee74aa32dd4c02b28fc325b088df Oct 30 21:47:12.680985 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:12.680955 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-phjxr" event={"ID":"753e9e9a-5c68-4540-8fc7-858514ca428e","Type":"ContainerStarted","Data":"5915fb03d92992b0a79d253956e0c461eb97ee74aa32dd4c02b28fc325b088df"} Oct 30 21:47:13.686599 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:13.686562 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-phjxr" event={"ID":"753e9e9a-5c68-4540-8fc7-858514ca428e","Type":"ContainerStarted","Data":"13baef02c23bd6c4155a404793df96f5b08e9673639a0f68f968a64a8c75266d"} Oct 30 21:47:13.686993 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:13.686644 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-phjxr" Oct 30 21:47:13.711677 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:13.711629 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-phjxr" podStartSLOduration=2.300010043 podStartE2EDuration="2.711614636s" totalImagesPullingTime="411.604593ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:13.710475108 +0000 UTC m=+399.881440286" watchObservedRunningTime="2026-10-30 21:47:13.711614636 +0000 UTC m=+399.882579817" Oct 30 21:47:18.286148 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:18.286112 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:18.377699 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:18.377665 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5cbf66d55-5wjvj"] Oct 30 21:47:18.377903 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:18.377881 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5cbf66d55-5wjvj" podUID="d0a2517a-f83a-42e1-8078-34eed5777c3b" containerName="maas-api" containerID="cri-o://eb3e30486a245ddf9437df1886a4081cb7c2c35f9f93364123e310cd8e122a84" gracePeriod=30 Oct 30 21:47:18.436140 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:18.436111 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:18.631713 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:18.631687 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5cbf66d55-5wjvj" Oct 30 21:47:18.710745 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:18.710709 6720 generic.go:356] "Generic (PLEG): container finished" podID="d0a2517a-f83a-42e1-8078-34eed5777c3b" containerID="eb3e30486a245ddf9437df1886a4081cb7c2c35f9f93364123e310cd8e122a84" exitCode=0 Oct 30 21:47:18.710908 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:18.710763 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5cbf66d55-5wjvj" Oct 30 21:47:18.710908 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:18.710801 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5cbf66d55-5wjvj" event={"ID":"d0a2517a-f83a-42e1-8078-34eed5777c3b","Type":"ContainerDied","Data":"eb3e30486a245ddf9437df1886a4081cb7c2c35f9f93364123e310cd8e122a84"} Oct 30 21:47:18.710908 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:18.710855 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5cbf66d55-5wjvj" event={"ID":"d0a2517a-f83a-42e1-8078-34eed5777c3b","Type":"ContainerDied","Data":"05c56cb6c834fce6559df02a5d72264b534ae4b2697df1de9db7fe8fdd057b20"} Oct 30 21:47:18.710908 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:18.710882 6720 scope.go:122] "RemoveContainer" containerID="eb3e30486a245ddf9437df1886a4081cb7c2c35f9f93364123e310cd8e122a84" Oct 30 21:47:18.719959 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:18.719942 6720 scope.go:122] "RemoveContainer" containerID="eb3e30486a245ddf9437df1886a4081cb7c2c35f9f93364123e310cd8e122a84" Oct 30 21:47:18.720219 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:47:18.720202 6720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb3e30486a245ddf9437df1886a4081cb7c2c35f9f93364123e310cd8e122a84\": container with ID starting with eb3e30486a245ddf9437df1886a4081cb7c2c35f9f93364123e310cd8e122a84 not found: ID does not exist" containerID="eb3e30486a245ddf9437df1886a4081cb7c2c35f9f93364123e310cd8e122a84" Oct 30 21:47:18.720274 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:18.720234 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb3e30486a245ddf9437df1886a4081cb7c2c35f9f93364123e310cd8e122a84"} err="failed to get container status \"eb3e30486a245ddf9437df1886a4081cb7c2c35f9f93364123e310cd8e122a84\": rpc error: code = NotFound desc = could not find container \"eb3e30486a245ddf9437df1886a4081cb7c2c35f9f93364123e310cd8e122a84\": container with ID starting with eb3e30486a245ddf9437df1886a4081cb7c2c35f9f93364123e310cd8e122a84 not found: ID does not exist" Oct 30 21:47:18.722599 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:18.722577 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0a2517a-f83a-42e1-8078-34eed5777c3b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0a2517a-f83a-42e1-8078-34eed5777c3b-metrics-certs\") pod \"d0a2517a-f83a-42e1-8078-34eed5777c3b\" (UID: \"d0a2517a-f83a-42e1-8078-34eed5777c3b\") " Oct 30 21:47:18.722707 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:18.722638 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0a2517a-f83a-42e1-8078-34eed5777c3b-kube-api-access-vvxdk\" (UniqueName: \"kubernetes.io/projected/d0a2517a-f83a-42e1-8078-34eed5777c3b-kube-api-access-vvxdk\") pod \"d0a2517a-f83a-42e1-8078-34eed5777c3b\" (UID: \"d0a2517a-f83a-42e1-8078-34eed5777c3b\") " Oct 30 21:47:18.722788 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:18.722773 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0a2517a-f83a-42e1-8078-34eed5777c3b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0a2517a-f83a-42e1-8078-34eed5777c3b-maas-api-tls\") pod \"d0a2517a-f83a-42e1-8078-34eed5777c3b\" (UID: \"d0a2517a-f83a-42e1-8078-34eed5777c3b\") " Oct 30 21:47:18.722842 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:18.722812 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0a2517a-f83a-42e1-8078-34eed5777c3b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0a2517a-f83a-42e1-8078-34eed5777c3b-openshift-service-ca\") pod \"d0a2517a-f83a-42e1-8078-34eed5777c3b\" (UID: \"d0a2517a-f83a-42e1-8078-34eed5777c3b\") " Oct 30 21:47:18.723225 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:18.723156 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0a2517a-f83a-42e1-8078-34eed5777c3b-openshift-service-ca" pod "d0a2517a-f83a-42e1-8078-34eed5777c3b" (UID: "d0a2517a-f83a-42e1-8078-34eed5777c3b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:47:18.724824 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:18.724804 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0a2517a-f83a-42e1-8078-34eed5777c3b-metrics-certs" pod "d0a2517a-f83a-42e1-8078-34eed5777c3b" (UID: "d0a2517a-f83a-42e1-8078-34eed5777c3b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:47:18.725006 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:18.724989 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0a2517a-f83a-42e1-8078-34eed5777c3b-maas-api-tls" pod "d0a2517a-f83a-42e1-8078-34eed5777c3b" (UID: "d0a2517a-f83a-42e1-8078-34eed5777c3b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:47:18.725093 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:18.725033 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0a2517a-f83a-42e1-8078-34eed5777c3b-kube-api-access-vvxdk" pod "d0a2517a-f83a-42e1-8078-34eed5777c3b" (UID: "d0a2517a-f83a-42e1-8078-34eed5777c3b"). InnerVolumeSpecName "kube-api-access-vvxdk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:47:18.823685 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:18.823620 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvxdk\" (UniqueName: \"kubernetes.io/projected/d0a2517a-f83a-42e1-8078-34eed5777c3b-kube-api-access-vvxdk\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:18.823685 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:18.823647 6720 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0a2517a-f83a-42e1-8078-34eed5777c3b-maas-api-tls\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:18.823685 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:18.823658 6720 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0a2517a-f83a-42e1-8078-34eed5777c3b-openshift-service-ca\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:18.823685 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:18.823667 6720 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0a2517a-f83a-42e1-8078-34eed5777c3b-metrics-certs\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:19.033204 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:19.033175 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5cbf66d55-5wjvj"] Oct 30 21:47:19.037366 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:19.037341 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5cbf66d55-5wjvj"] Oct 30 21:47:19.956328 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:19.956026 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0a2517a-f83a-42e1-8078-34eed5777c3b" path="/var/lib/kubelet/pods/d0a2517a-f83a-42e1-8078-34eed5777c3b/volumes" Oct 30 21:47:25.893119 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:25.893088 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:32.416945 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:32.416909 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:32.566212 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:32.566173 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:40.401480 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:47:40.401439 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 21:47:44.692908 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:44.692875 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-phjxr" Oct 30 21:47:44.741788 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:44.741756 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-k2lk6"] Oct 30 21:47:44.829899 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:44.829866 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-k2lk6" podUID="7c9a9adc-e96d-4b46-9399-04a8cdb7eb1c" containerName="manager" containerID="cri-o://80f8b4eaf55c26fcee7217b9720a3a610e46b04a789f73e4ae614bfa9a9926ec" gracePeriod=30 Oct 30 21:47:45.069819 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:45.069789 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-k2lk6" Oct 30 21:47:45.137051 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:45.137027 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c9a9adc-e96d-4b46-9399-04a8cdb7eb1c-cert\" (UniqueName: \"kubernetes.io/secret/7c9a9adc-e96d-4b46-9399-04a8cdb7eb1c-cert\") pod \"7c9a9adc-e96d-4b46-9399-04a8cdb7eb1c\" (UID: \"7c9a9adc-e96d-4b46-9399-04a8cdb7eb1c\") " Oct 30 21:47:45.137210 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:45.137111 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c9a9adc-e96d-4b46-9399-04a8cdb7eb1c-kube-api-access-kskhr\" (UniqueName: \"kubernetes.io/projected/7c9a9adc-e96d-4b46-9399-04a8cdb7eb1c-kube-api-access-kskhr\") pod \"7c9a9adc-e96d-4b46-9399-04a8cdb7eb1c\" (UID: \"7c9a9adc-e96d-4b46-9399-04a8cdb7eb1c\") " Oct 30 21:47:45.139184 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:45.139144 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c9a9adc-e96d-4b46-9399-04a8cdb7eb1c-cert" pod "7c9a9adc-e96d-4b46-9399-04a8cdb7eb1c" (UID: "7c9a9adc-e96d-4b46-9399-04a8cdb7eb1c"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:47:45.139285 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:45.139234 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c9a9adc-e96d-4b46-9399-04a8cdb7eb1c-kube-api-access-kskhr" pod "7c9a9adc-e96d-4b46-9399-04a8cdb7eb1c" (UID: "7c9a9adc-e96d-4b46-9399-04a8cdb7eb1c"). InnerVolumeSpecName "kube-api-access-kskhr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:47:45.237608 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:45.237585 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kskhr\" (UniqueName: \"kubernetes.io/projected/7c9a9adc-e96d-4b46-9399-04a8cdb7eb1c-kube-api-access-kskhr\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:45.237608 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:45.237608 6720 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7c9a9adc-e96d-4b46-9399-04a8cdb7eb1c-cert\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:47:45.836273 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:45.836237 6720 generic.go:356] "Generic (PLEG): container finished" podID="7c9a9adc-e96d-4b46-9399-04a8cdb7eb1c" containerID="80f8b4eaf55c26fcee7217b9720a3a610e46b04a789f73e4ae614bfa9a9926ec" exitCode=0 Oct 30 21:47:45.836720 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:45.836290 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-k2lk6" Oct 30 21:47:45.836720 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:45.836326 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-k2lk6" event={"ID":"7c9a9adc-e96d-4b46-9399-04a8cdb7eb1c","Type":"ContainerDied","Data":"80f8b4eaf55c26fcee7217b9720a3a610e46b04a789f73e4ae614bfa9a9926ec"} Oct 30 21:47:45.836720 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:45.836398 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-k2lk6" event={"ID":"7c9a9adc-e96d-4b46-9399-04a8cdb7eb1c","Type":"ContainerDied","Data":"0ba4549b929f6012408faa970edf5bd5d1ed936cee8b1c146fd1e5187df5ec6a"} Oct 30 21:47:45.836720 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:45.836409 6720 scope.go:122] "RemoveContainer" containerID="80f8b4eaf55c26fcee7217b9720a3a610e46b04a789f73e4ae614bfa9a9926ec" Oct 30 21:47:45.846114 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:45.846097 6720 scope.go:122] "RemoveContainer" containerID="80f8b4eaf55c26fcee7217b9720a3a610e46b04a789f73e4ae614bfa9a9926ec" Oct 30 21:47:45.846364 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:47:45.846346 6720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80f8b4eaf55c26fcee7217b9720a3a610e46b04a789f73e4ae614bfa9a9926ec\": container with ID starting with 80f8b4eaf55c26fcee7217b9720a3a610e46b04a789f73e4ae614bfa9a9926ec not found: ID does not exist" containerID="80f8b4eaf55c26fcee7217b9720a3a610e46b04a789f73e4ae614bfa9a9926ec" Oct 30 21:47:45.846441 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:45.846372 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80f8b4eaf55c26fcee7217b9720a3a610e46b04a789f73e4ae614bfa9a9926ec"} err="failed to get container status \"80f8b4eaf55c26fcee7217b9720a3a610e46b04a789f73e4ae614bfa9a9926ec\": rpc error: code = NotFound desc = could not find container \"80f8b4eaf55c26fcee7217b9720a3a610e46b04a789f73e4ae614bfa9a9926ec\": container with ID starting with 80f8b4eaf55c26fcee7217b9720a3a610e46b04a789f73e4ae614bfa9a9926ec not found: ID does not exist" Oct 30 21:47:45.859510 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:45.859487 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-k2lk6"] Oct 30 21:47:45.865650 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:45.865630 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-k2lk6"] Oct 30 21:47:45.942513 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:45.942479 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:45.954826 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:45.954793 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c9a9adc-e96d-4b46-9399-04a8cdb7eb1c" path="/var/lib/kubelet/pods/7c9a9adc-e96d-4b46-9399-04a8cdb7eb1c/volumes" Oct 30 21:47:46.124617 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:46.124586 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:49.623521 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:49.623484 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:47:49.862621 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:47:49.862589 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:08.839988 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:08.839949 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:08.987236 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:08.987204 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:10.401055 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:48:10.401021 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 21:48:16.394265 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:16.394185 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:16.936808 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:16.936768 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:18.262152 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:18.262121 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:18.434038 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:18.434008 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:21.801634 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:21.801596 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:25.230025 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:25.229989 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:25.981378 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:25.981337 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:26.477005 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:26.476968 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:27.377065 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:27.377034 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:28.518038 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:28.518001 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:31.678237 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:31.678202 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:33.006703 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:33.006666 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:35.373077 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:35.373045 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:40.402166 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:48:40.402124 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 21:48:48.499723 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:48.499688 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-87c457c84-q9js5"] Oct 30 21:48:48.500098 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:48.500049 6720 cpu_manager.go:414] "containerMap: removing container" podUID="d0a2517a-f83a-42e1-8078-34eed5777c3b" containerName="maas-api" Oct 30 21:48:48.500098 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:48.500061 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0a2517a-f83a-42e1-8078-34eed5777c3b" containerName="maas-api" Oct 30 21:48:48.500098 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:48.500077 6720 cpu_manager.go:414] "containerMap: removing container" podUID="7c9a9adc-e96d-4b46-9399-04a8cdb7eb1c" containerName="manager" Oct 30 21:48:48.500098 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:48.500082 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c9a9adc-e96d-4b46-9399-04a8cdb7eb1c" containerName="manager" Oct 30 21:48:48.500228 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:48.500161 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0a2517a-f83a-42e1-8078-34eed5777c3b" containerName="maas-api" Oct 30 21:48:48.500228 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:48.500170 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c9a9adc-e96d-4b46-9399-04a8cdb7eb1c" containerName="manager" Oct 30 21:48:48.503053 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:48.503031 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-87c457c84-q9js5" Oct 30 21:48:48.513056 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:48.513031 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-87c457c84-q9js5"] Oct 30 21:48:48.592616 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:48.592583 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bd0b4436-451d-4abd-aba0-3ebb841ba48e-tls-cert\") pod \"authorino-87c457c84-q9js5\" (UID: \"bd0b4436-451d-4abd-aba0-3ebb841ba48e\") " pod="kuadrant-system/authorino-87c457c84-q9js5" Oct 30 21:48:48.592768 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:48.592688 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqb8d\" (UniqueName: \"kubernetes.io/projected/bd0b4436-451d-4abd-aba0-3ebb841ba48e-kube-api-access-nqb8d\") pod \"authorino-87c457c84-q9js5\" (UID: \"bd0b4436-451d-4abd-aba0-3ebb841ba48e\") " pod="kuadrant-system/authorino-87c457c84-q9js5" Oct 30 21:48:48.694062 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:48.694029 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bd0b4436-451d-4abd-aba0-3ebb841ba48e-tls-cert\") pod \"authorino-87c457c84-q9js5\" (UID: \"bd0b4436-451d-4abd-aba0-3ebb841ba48e\") " pod="kuadrant-system/authorino-87c457c84-q9js5" Oct 30 21:48:48.694207 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:48.694083 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqb8d\" (UniqueName: \"kubernetes.io/projected/bd0b4436-451d-4abd-aba0-3ebb841ba48e-kube-api-access-nqb8d\") pod \"authorino-87c457c84-q9js5\" (UID: \"bd0b4436-451d-4abd-aba0-3ebb841ba48e\") " pod="kuadrant-system/authorino-87c457c84-q9js5" Oct 30 21:48:48.696714 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:48.696687 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bd0b4436-451d-4abd-aba0-3ebb841ba48e-tls-cert\") pod \"authorino-87c457c84-q9js5\" (UID: \"bd0b4436-451d-4abd-aba0-3ebb841ba48e\") " pod="kuadrant-system/authorino-87c457c84-q9js5" Oct 30 21:48:48.709291 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:48.709262 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqb8d\" (UniqueName: \"kubernetes.io/projected/bd0b4436-451d-4abd-aba0-3ebb841ba48e-kube-api-access-nqb8d\") pod \"authorino-87c457c84-q9js5\" (UID: \"bd0b4436-451d-4abd-aba0-3ebb841ba48e\") " pod="kuadrant-system/authorino-87c457c84-q9js5" Oct 30 21:48:48.813978 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:48.813916 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-87c457c84-q9js5" Oct 30 21:48:49.086531 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:49.086412 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-87c457c84-q9js5"] Oct 30 21:48:49.089511 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:48:49.089484 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbd0b4436_451d_4abd_aba0_3ebb841ba48e.slice/crio-40c30e0f0b0c0a304f02daeeb588f51370c7d98f172175cea3d5075279e200cb WatchSource:0}: Error finding container 40c30e0f0b0c0a304f02daeeb588f51370c7d98f172175cea3d5075279e200cb: Status 404 returned error can't find the container with id 40c30e0f0b0c0a304f02daeeb588f51370c7d98f172175cea3d5075279e200cb Oct 30 21:48:49.128904 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:49.128878 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-87c457c84-q9js5" event={"ID":"bd0b4436-451d-4abd-aba0-3ebb841ba48e","Type":"ContainerStarted","Data":"40c30e0f0b0c0a304f02daeeb588f51370c7d98f172175cea3d5075279e200cb"} Oct 30 21:48:49.305110 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:49.305083 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:50.136568 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:50.136527 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-87c457c84-q9js5" event={"ID":"bd0b4436-451d-4abd-aba0-3ebb841ba48e","Type":"ContainerStarted","Data":"ea5524fbb66cefb17a3b30fbc5cfa37cb4e408411a14add079170152808214d3"} Oct 30 21:48:50.155641 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:50.155584 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-87c457c84-q9js5" podStartSLOduration=1.448986693 podStartE2EDuration="2.155569943s" totalImagesPullingTime="706.58325ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:50.154407045 +0000 UTC m=+496.325372226" watchObservedRunningTime="2026-10-30 21:48:50.155569943 +0000 UTC m=+496.326535125" Oct 30 21:48:50.188778 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:50.188742 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-f77ccdd9f-2w568"] Oct 30 21:48:50.188984 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:50.188931 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-f77ccdd9f-2w568" podUID="9476a84f-8210-47db-a3f1-66f5a5eea2db" containerName="authorino" containerID="cri-o://2f4556627b5c0071e9652048eb78e1fa3bb72e31d690dd1a6fe7ed3b4e760398" gracePeriod=30 Oct 30 21:48:50.459835 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:50.459810 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-f77ccdd9f-2w568" Oct 30 21:48:50.510817 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:50.510788 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9476a84f-8210-47db-a3f1-66f5a5eea2db-kube-api-access-h826z\" (UniqueName: \"kubernetes.io/projected/9476a84f-8210-47db-a3f1-66f5a5eea2db-kube-api-access-h826z\") pod \"9476a84f-8210-47db-a3f1-66f5a5eea2db\" (UID: \"9476a84f-8210-47db-a3f1-66f5a5eea2db\") " Oct 30 21:48:50.511000 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:50.510905 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9476a84f-8210-47db-a3f1-66f5a5eea2db-tls-cert\" (UniqueName: \"kubernetes.io/secret/9476a84f-8210-47db-a3f1-66f5a5eea2db-tls-cert\") pod \"9476a84f-8210-47db-a3f1-66f5a5eea2db\" (UID: \"9476a84f-8210-47db-a3f1-66f5a5eea2db\") " Oct 30 21:48:50.513155 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:50.513124 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9476a84f-8210-47db-a3f1-66f5a5eea2db-kube-api-access-h826z" pod "9476a84f-8210-47db-a3f1-66f5a5eea2db" (UID: "9476a84f-8210-47db-a3f1-66f5a5eea2db"). InnerVolumeSpecName "kube-api-access-h826z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:48:50.521608 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:50.521582 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9476a84f-8210-47db-a3f1-66f5a5eea2db-tls-cert" pod "9476a84f-8210-47db-a3f1-66f5a5eea2db" (UID: "9476a84f-8210-47db-a3f1-66f5a5eea2db"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:48:50.611956 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:50.611919 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h826z\" (UniqueName: \"kubernetes.io/projected/9476a84f-8210-47db-a3f1-66f5a5eea2db-kube-api-access-h826z\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:50.611956 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:50.611952 6720 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9476a84f-8210-47db-a3f1-66f5a5eea2db-tls-cert\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:48:51.141964 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:51.141931 6720 generic.go:356] "Generic (PLEG): container finished" podID="9476a84f-8210-47db-a3f1-66f5a5eea2db" containerID="2f4556627b5c0071e9652048eb78e1fa3bb72e31d690dd1a6fe7ed3b4e760398" exitCode=0 Oct 30 21:48:51.142456 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:51.141995 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-f77ccdd9f-2w568" Oct 30 21:48:51.142456 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:51.142025 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-f77ccdd9f-2w568" event={"ID":"9476a84f-8210-47db-a3f1-66f5a5eea2db","Type":"ContainerDied","Data":"2f4556627b5c0071e9652048eb78e1fa3bb72e31d690dd1a6fe7ed3b4e760398"} Oct 30 21:48:51.142456 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:51.142076 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-f77ccdd9f-2w568" event={"ID":"9476a84f-8210-47db-a3f1-66f5a5eea2db","Type":"ContainerDied","Data":"d2bb34f2e3bea346f3dcef024b1674bea836cb9147f32a366aad70a747ef52ed"} Oct 30 21:48:51.142456 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:51.142101 6720 scope.go:122] "RemoveContainer" containerID="2f4556627b5c0071e9652048eb78e1fa3bb72e31d690dd1a6fe7ed3b4e760398" Oct 30 21:48:51.152094 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:51.152071 6720 scope.go:122] "RemoveContainer" containerID="2f4556627b5c0071e9652048eb78e1fa3bb72e31d690dd1a6fe7ed3b4e760398" Oct 30 21:48:51.152420 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:48:51.152394 6720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f4556627b5c0071e9652048eb78e1fa3bb72e31d690dd1a6fe7ed3b4e760398\": container with ID starting with 2f4556627b5c0071e9652048eb78e1fa3bb72e31d690dd1a6fe7ed3b4e760398 not found: ID does not exist" containerID="2f4556627b5c0071e9652048eb78e1fa3bb72e31d690dd1a6fe7ed3b4e760398" Oct 30 21:48:51.152502 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:51.152433 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f4556627b5c0071e9652048eb78e1fa3bb72e31d690dd1a6fe7ed3b4e760398"} err="failed to get container status \"2f4556627b5c0071e9652048eb78e1fa3bb72e31d690dd1a6fe7ed3b4e760398\": rpc error: code = NotFound desc = could not find container \"2f4556627b5c0071e9652048eb78e1fa3bb72e31d690dd1a6fe7ed3b4e760398\": container with ID starting with 2f4556627b5c0071e9652048eb78e1fa3bb72e31d690dd1a6fe7ed3b4e760398 not found: ID does not exist" Oct 30 21:48:51.167507 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:51.167481 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-f77ccdd9f-2w568"] Oct 30 21:48:51.172909 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:51.172885 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-f77ccdd9f-2w568"] Oct 30 21:48:51.345313 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:51.345281 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:51.561698 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:51.561609 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:51.955376 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:51.955344 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9476a84f-8210-47db-a3f1-66f5a5eea2db" path="/var/lib/kubelet/pods/9476a84f-8210-47db-a3f1-66f5a5eea2db/volumes" Oct 30 21:48:53.062318 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:53.062274 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:53.285097 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:53.285051 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:48:53.984703 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:53.984670 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sqg22"] Oct 30 21:48:53.985279 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:53.985255 6720 cpu_manager.go:414] "containerMap: removing container" podUID="9476a84f-8210-47db-a3f1-66f5a5eea2db" containerName="authorino" Oct 30 21:48:53.985402 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:53.985281 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9476a84f-8210-47db-a3f1-66f5a5eea2db" containerName="authorino" Oct 30 21:48:53.985489 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:53.985431 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="9476a84f-8210-47db-a3f1-66f5a5eea2db" containerName="authorino" Oct 30 21:48:53.989794 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:53.989766 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sqg22" Oct 30 21:48:54.015442 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:54.015412 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-kx6b6\"" Oct 30 21:48:54.025134 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:54.025109 6720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 21:48:54.029288 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:54.029266 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sqg22"] Oct 30 21:48:54.090825 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:54.090791 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrsfj\" (UniqueName: \"kubernetes.io/projected/ccc62572-e0cd-43fb-8fc2-7715f2223da3-kube-api-access-qrsfj\") pod \"maas-default-gateway-openshift-default-85544d4cd7-sqg22\" (UID: \"ccc62572-e0cd-43fb-8fc2-7715f2223da3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sqg22" Oct 30 21:48:54.090825 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:54.090824 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccc62572-e0cd-43fb-8fc2-7715f2223da3-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-sqg22\" (UID: \"ccc62572-e0cd-43fb-8fc2-7715f2223da3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sqg22" Oct 30 21:48:54.091290 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:54.090843 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccc62572-e0cd-43fb-8fc2-7715f2223da3-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-sqg22\" (UID: \"ccc62572-e0cd-43fb-8fc2-7715f2223da3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sqg22" Oct 30 21:48:54.091290 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:54.090865 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccc62572-e0cd-43fb-8fc2-7715f2223da3-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-sqg22\" (UID: \"ccc62572-e0cd-43fb-8fc2-7715f2223da3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sqg22" Oct 30 21:48:54.091290 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:54.090975 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccc62572-e0cd-43fb-8fc2-7715f2223da3-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-sqg22\" (UID: \"ccc62572-e0cd-43fb-8fc2-7715f2223da3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sqg22" Oct 30 21:48:54.091290 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:54.091009 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccc62572-e0cd-43fb-8fc2-7715f2223da3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-sqg22\" (UID: \"ccc62572-e0cd-43fb-8fc2-7715f2223da3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sqg22" Oct 30 21:48:54.091290 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:54.091084 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccc62572-e0cd-43fb-8fc2-7715f2223da3-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-sqg22\" (UID: \"ccc62572-e0cd-43fb-8fc2-7715f2223da3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sqg22" Oct 30 21:48:54.091290 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:54.091140 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccc62572-e0cd-43fb-8fc2-7715f2223da3-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-sqg22\" (UID: \"ccc62572-e0cd-43fb-8fc2-7715f2223da3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sqg22" Oct 30 21:48:54.091290 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:54.091178 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccc62572-e0cd-43fb-8fc2-7715f2223da3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-sqg22\" (UID: \"ccc62572-e0cd-43fb-8fc2-7715f2223da3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sqg22" Oct 30 21:48:54.191803 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:54.191763 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrsfj\" (UniqueName: \"kubernetes.io/projected/ccc62572-e0cd-43fb-8fc2-7715f2223da3-kube-api-access-qrsfj\") pod \"maas-default-gateway-openshift-default-85544d4cd7-sqg22\" (UID: \"ccc62572-e0cd-43fb-8fc2-7715f2223da3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sqg22" Oct 30 21:48:54.191985 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:54.191817 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccc62572-e0cd-43fb-8fc2-7715f2223da3-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-sqg22\" (UID: \"ccc62572-e0cd-43fb-8fc2-7715f2223da3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sqg22" Oct 30 21:48:54.191985 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:54.191835 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccc62572-e0cd-43fb-8fc2-7715f2223da3-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-sqg22\" (UID: \"ccc62572-e0cd-43fb-8fc2-7715f2223da3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sqg22" Oct 30 21:48:54.191985 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:54.191862 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccc62572-e0cd-43fb-8fc2-7715f2223da3-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-sqg22\" (UID: \"ccc62572-e0cd-43fb-8fc2-7715f2223da3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sqg22" Oct 30 21:48:54.191985 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:54.191887 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccc62572-e0cd-43fb-8fc2-7715f2223da3-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-sqg22\" (UID: \"ccc62572-e0cd-43fb-8fc2-7715f2223da3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sqg22" Oct 30 21:48:54.191985 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:54.191907 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccc62572-e0cd-43fb-8fc2-7715f2223da3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-sqg22\" (UID: \"ccc62572-e0cd-43fb-8fc2-7715f2223da3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sqg22" Oct 30 21:48:54.191985 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:54.191956 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccc62572-e0cd-43fb-8fc2-7715f2223da3-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-sqg22\" (UID: \"ccc62572-e0cd-43fb-8fc2-7715f2223da3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sqg22" Oct 30 21:48:54.192365 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:54.192021 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccc62572-e0cd-43fb-8fc2-7715f2223da3-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-sqg22\" (UID: \"ccc62572-e0cd-43fb-8fc2-7715f2223da3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sqg22" Oct 30 21:48:54.192365 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:54.192059 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccc62572-e0cd-43fb-8fc2-7715f2223da3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-sqg22\" (UID: \"ccc62572-e0cd-43fb-8fc2-7715f2223da3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sqg22" Oct 30 21:48:54.192503 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:54.192367 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccc62572-e0cd-43fb-8fc2-7715f2223da3-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-sqg22\" (UID: \"ccc62572-e0cd-43fb-8fc2-7715f2223da3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sqg22" Oct 30 21:48:54.192503 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:54.192449 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccc62572-e0cd-43fb-8fc2-7715f2223da3-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-sqg22\" (UID: \"ccc62572-e0cd-43fb-8fc2-7715f2223da3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sqg22" Oct 30 21:48:54.192612 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:54.192564 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccc62572-e0cd-43fb-8fc2-7715f2223da3-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-sqg22\" (UID: \"ccc62572-e0cd-43fb-8fc2-7715f2223da3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sqg22" Oct 30 21:48:54.192669 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:54.192648 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccc62572-e0cd-43fb-8fc2-7715f2223da3-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-sqg22\" (UID: \"ccc62572-e0cd-43fb-8fc2-7715f2223da3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sqg22" Oct 30 21:48:54.192866 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:54.192840 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccc62572-e0cd-43fb-8fc2-7715f2223da3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-sqg22\" (UID: \"ccc62572-e0cd-43fb-8fc2-7715f2223da3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sqg22" Oct 30 21:48:54.194611 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:54.194588 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccc62572-e0cd-43fb-8fc2-7715f2223da3-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-sqg22\" (UID: \"ccc62572-e0cd-43fb-8fc2-7715f2223da3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sqg22" Oct 30 21:48:54.194716 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:54.194632 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccc62572-e0cd-43fb-8fc2-7715f2223da3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-sqg22\" (UID: \"ccc62572-e0cd-43fb-8fc2-7715f2223da3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sqg22" Oct 30 21:48:54.223349 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:54.223318 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccc62572-e0cd-43fb-8fc2-7715f2223da3-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-sqg22\" (UID: \"ccc62572-e0cd-43fb-8fc2-7715f2223da3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sqg22" Oct 30 21:48:54.243350 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:54.243278 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrsfj\" (UniqueName: \"kubernetes.io/projected/ccc62572-e0cd-43fb-8fc2-7715f2223da3-kube-api-access-qrsfj\") pod \"maas-default-gateway-openshift-default-85544d4cd7-sqg22\" (UID: \"ccc62572-e0cd-43fb-8fc2-7715f2223da3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sqg22" Oct 30 21:48:54.307136 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:54.307098 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sqg22" Oct 30 21:48:54.589697 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:54.589618 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sqg22"] Oct 30 21:48:54.593107 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:48:54.593081 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccc62572_e0cd_43fb_8fc2_7715f2223da3.slice/crio-8fa31333f23f8f3e45b267657fa808c9005a01188d3bfc178e1d4a2d08313c46 WatchSource:0}: Error finding container 8fa31333f23f8f3e45b267657fa808c9005a01188d3bfc178e1d4a2d08313c46: Status 404 returned error can't find the container with id 8fa31333f23f8f3e45b267657fa808c9005a01188d3bfc178e1d4a2d08313c46 Oct 30 21:48:54.596026 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:54.596001 6720 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:48:55.164612 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:55.164575 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sqg22" event={"ID":"ccc62572-e0cd-43fb-8fc2-7715f2223da3","Type":"ContainerStarted","Data":"8fa31333f23f8f3e45b267657fa808c9005a01188d3bfc178e1d4a2d08313c46"} Oct 30 21:48:57.579964 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:57.579929 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:48:57.580251 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:57.579996 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:48:57.580251 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:57.580024 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:48:58.183352 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:58.183314 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sqg22" event={"ID":"ccc62572-e0cd-43fb-8fc2-7715f2223da3","Type":"ContainerStarted","Data":"826e74d0c65ce2d651d89e86cb5483a1663ac68b1ff5907ce0f605eb1b033b59"} Oct 30 21:48:58.215199 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:58.215149 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sqg22" podStartSLOduration=2.232466599 podStartE2EDuration="5.215131428s" totalImagesPullingTime="2.982664829s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:58.214631389 +0000 UTC m=+504.385596573" watchObservedRunningTime="2026-10-30 21:48:58.215131428 +0000 UTC m=+504.386096610" Oct 30 21:48:58.308012 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:58.307970 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sqg22" Oct 30 21:48:58.309561 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:58.309522 6720 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-sqg22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 30 21:48:58.309628 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:58.309592 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sqg22" podUID="ccc62572-e0cd-43fb-8fc2-7715f2223da3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 30 21:48:59.307484 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:59.307448 6720 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-sqg22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 30 21:48:59.307867 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:48:59.307520 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sqg22" podUID="ccc62572-e0cd-43fb-8fc2-7715f2223da3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 30 21:49:00.307516 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:00.307481 6720 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-sqg22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 30 21:49:00.307910 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:00.307537 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sqg22" podUID="ccc62572-e0cd-43fb-8fc2-7715f2223da3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 30 21:49:01.308157 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:01.308121 6720 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-sqg22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 30 21:49:01.308591 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:01.308192 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sqg22" podUID="ccc62572-e0cd-43fb-8fc2-7715f2223da3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 30 21:49:02.308330 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:02.308288 6720 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-sqg22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 30 21:49:02.308712 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:02.308348 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sqg22" podUID="ccc62572-e0cd-43fb-8fc2-7715f2223da3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 30 21:49:03.092286 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:03.092251 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:03.308331 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:03.308294 6720 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-sqg22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 30 21:49:03.308764 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:03.308356 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sqg22" podUID="ccc62572-e0cd-43fb-8fc2-7715f2223da3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 30 21:49:04.307805 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:04.307766 6720 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-sqg22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 30 21:49:04.307950 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:04.307841 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sqg22" podUID="ccc62572-e0cd-43fb-8fc2-7715f2223da3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 30 21:49:05.767573 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:05.767542 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sqg22" Oct 30 21:49:05.768051 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:05.767945 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sqg22" Oct 30 21:49:05.768961 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:05.768941 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-sqg22" Oct 30 21:49:08.995724 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:08.995680 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grjn"] Oct 30 21:49:08.999580 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:08.999562 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grjn" Oct 30 21:49:09.015440 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:09.015413 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grjn"] Oct 30 21:49:09.062635 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:09.062595 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3dfec03-d265-405d-bc13-fcf8f44b794d-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8grjn\" (UID: \"b3dfec03-d265-405d-bc13-fcf8f44b794d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grjn" Oct 30 21:49:09.062635 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:09.062634 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3dfec03-d265-405d-bc13-fcf8f44b794d-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8grjn\" (UID: \"b3dfec03-d265-405d-bc13-fcf8f44b794d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grjn" Oct 30 21:49:09.062850 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:09.062661 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3dfec03-d265-405d-bc13-fcf8f44b794d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8grjn\" (UID: \"b3dfec03-d265-405d-bc13-fcf8f44b794d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grjn" Oct 30 21:49:09.062850 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:09.062807 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3dfec03-d265-405d-bc13-fcf8f44b794d-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8grjn\" (UID: \"b3dfec03-d265-405d-bc13-fcf8f44b794d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grjn" Oct 30 21:49:09.062924 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:09.062849 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3dfec03-d265-405d-bc13-fcf8f44b794d-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8grjn\" (UID: \"b3dfec03-d265-405d-bc13-fcf8f44b794d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grjn" Oct 30 21:49:09.062967 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:09.062940 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lngxg\" (UniqueName: \"kubernetes.io/projected/b3dfec03-d265-405d-bc13-fcf8f44b794d-kube-api-access-lngxg\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8grjn\" (UID: \"b3dfec03-d265-405d-bc13-fcf8f44b794d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grjn" Oct 30 21:49:09.063003 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:09.062989 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3dfec03-d265-405d-bc13-fcf8f44b794d-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8grjn\" (UID: \"b3dfec03-d265-405d-bc13-fcf8f44b794d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grjn" Oct 30 21:49:09.063042 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:09.063010 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3dfec03-d265-405d-bc13-fcf8f44b794d-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8grjn\" (UID: \"b3dfec03-d265-405d-bc13-fcf8f44b794d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grjn" Oct 30 21:49:09.063076 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:09.063046 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3dfec03-d265-405d-bc13-fcf8f44b794d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8grjn\" (UID: \"b3dfec03-d265-405d-bc13-fcf8f44b794d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grjn" Oct 30 21:49:09.163786 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:09.163753 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3dfec03-d265-405d-bc13-fcf8f44b794d-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8grjn\" (UID: \"b3dfec03-d265-405d-bc13-fcf8f44b794d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grjn" Oct 30 21:49:09.163968 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:09.163795 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3dfec03-d265-405d-bc13-fcf8f44b794d-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8grjn\" (UID: \"b3dfec03-d265-405d-bc13-fcf8f44b794d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grjn" Oct 30 21:49:09.163968 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:09.163831 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lngxg\" (UniqueName: \"kubernetes.io/projected/b3dfec03-d265-405d-bc13-fcf8f44b794d-kube-api-access-lngxg\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8grjn\" (UID: \"b3dfec03-d265-405d-bc13-fcf8f44b794d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grjn" Oct 30 21:49:09.163968 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:09.163862 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3dfec03-d265-405d-bc13-fcf8f44b794d-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8grjn\" (UID: \"b3dfec03-d265-405d-bc13-fcf8f44b794d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grjn" Oct 30 21:49:09.163968 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:09.163876 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3dfec03-d265-405d-bc13-fcf8f44b794d-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8grjn\" (UID: \"b3dfec03-d265-405d-bc13-fcf8f44b794d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grjn" Oct 30 21:49:09.163968 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:09.163905 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3dfec03-d265-405d-bc13-fcf8f44b794d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8grjn\" (UID: \"b3dfec03-d265-405d-bc13-fcf8f44b794d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grjn" Oct 30 21:49:09.163968 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:09.163953 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3dfec03-d265-405d-bc13-fcf8f44b794d-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8grjn\" (UID: \"b3dfec03-d265-405d-bc13-fcf8f44b794d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grjn" Oct 30 21:49:09.164280 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:09.163986 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3dfec03-d265-405d-bc13-fcf8f44b794d-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8grjn\" (UID: \"b3dfec03-d265-405d-bc13-fcf8f44b794d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grjn" Oct 30 21:49:09.164280 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:09.164023 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3dfec03-d265-405d-bc13-fcf8f44b794d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8grjn\" (UID: \"b3dfec03-d265-405d-bc13-fcf8f44b794d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grjn" Oct 30 21:49:09.164280 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:09.164255 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3dfec03-d265-405d-bc13-fcf8f44b794d-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8grjn\" (UID: \"b3dfec03-d265-405d-bc13-fcf8f44b794d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grjn" Oct 30 21:49:09.164467 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:09.164294 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3dfec03-d265-405d-bc13-fcf8f44b794d-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8grjn\" (UID: \"b3dfec03-d265-405d-bc13-fcf8f44b794d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grjn" Oct 30 21:49:09.164467 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:09.164325 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3dfec03-d265-405d-bc13-fcf8f44b794d-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8grjn\" (UID: \"b3dfec03-d265-405d-bc13-fcf8f44b794d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grjn" Oct 30 21:49:09.164576 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:09.164527 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3dfec03-d265-405d-bc13-fcf8f44b794d-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8grjn\" (UID: \"b3dfec03-d265-405d-bc13-fcf8f44b794d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grjn" Oct 30 21:49:09.164978 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:09.164898 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3dfec03-d265-405d-bc13-fcf8f44b794d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8grjn\" (UID: \"b3dfec03-d265-405d-bc13-fcf8f44b794d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grjn" Oct 30 21:49:09.166558 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:09.166528 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3dfec03-d265-405d-bc13-fcf8f44b794d-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8grjn\" (UID: \"b3dfec03-d265-405d-bc13-fcf8f44b794d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grjn" Oct 30 21:49:09.166815 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:09.166798 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3dfec03-d265-405d-bc13-fcf8f44b794d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8grjn\" (UID: \"b3dfec03-d265-405d-bc13-fcf8f44b794d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grjn" Oct 30 21:49:09.177367 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:09.177343 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3dfec03-d265-405d-bc13-fcf8f44b794d-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8grjn\" (UID: \"b3dfec03-d265-405d-bc13-fcf8f44b794d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grjn" Oct 30 21:49:09.177516 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:09.177342 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lngxg\" (UniqueName: \"kubernetes.io/projected/b3dfec03-d265-405d-bc13-fcf8f44b794d-kube-api-access-lngxg\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8grjn\" (UID: \"b3dfec03-d265-405d-bc13-fcf8f44b794d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grjn" Oct 30 21:49:09.313576 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:09.313504 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grjn" Oct 30 21:49:09.595949 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:09.595879 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grjn"] Oct 30 21:49:09.599596 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:49:09.599573 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3dfec03_d265_405d_bc13_fcf8f44b794d.slice/crio-3ccb0541a2f4e7a43f627d78fff2baf0b788e388a0053d2a785a82c8f44c0f6a WatchSource:0}: Error finding container 3ccb0541a2f4e7a43f627d78fff2baf0b788e388a0053d2a785a82c8f44c0f6a: Status 404 returned error can't find the container with id 3ccb0541a2f4e7a43f627d78fff2baf0b788e388a0053d2a785a82c8f44c0f6a Oct 30 21:49:09.601742 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:09.601706 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:49:09.601822 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:09.601775 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:49:09.601822 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:09.601806 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:49:10.239679 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:10.239643 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grjn" event={"ID":"b3dfec03-d265-405d-bc13-fcf8f44b794d","Type":"ContainerStarted","Data":"888fbbd2232104633f0033c13604eac74e2ac09744527290175c24fdd7d645c2"} Oct 30 21:49:10.240055 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:10.239695 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grjn" event={"ID":"b3dfec03-d265-405d-bc13-fcf8f44b794d","Type":"ContainerStarted","Data":"3ccb0541a2f4e7a43f627d78fff2baf0b788e388a0053d2a785a82c8f44c0f6a"} Oct 30 21:49:10.285298 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:10.285246 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grjn" podStartSLOduration=2.285228676 podStartE2EDuration="2.285228676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:10.283506064 +0000 UTC m=+516.454471249" watchObservedRunningTime="2026-10-30 21:49:10.285228676 +0000 UTC m=+516.456193857" Oct 30 21:49:10.314235 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:10.314212 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grjn" Oct 30 21:49:10.315804 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:10.315777 6720 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-8grjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 30 21:49:10.315934 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:10.315824 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grjn" podUID="b3dfec03-d265-405d-bc13-fcf8f44b794d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 30 21:49:10.401848 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:49:10.401809 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 21:49:11.314288 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:11.314255 6720 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-8grjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 30 21:49:11.314802 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:11.314303 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grjn" podUID="b3dfec03-d265-405d-bc13-fcf8f44b794d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 30 21:49:12.314257 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:12.314222 6720 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-8grjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 30 21:49:12.314468 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:12.314276 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grjn" podUID="b3dfec03-d265-405d-bc13-fcf8f44b794d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 30 21:49:13.314323 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:13.314284 6720 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-8grjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 30 21:49:13.314524 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:13.314355 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grjn" podUID="b3dfec03-d265-405d-bc13-fcf8f44b794d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 30 21:49:14.314019 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:14.313989 6720 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-8grjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 30 21:49:14.314186 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:14.314049 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grjn" podUID="b3dfec03-d265-405d-bc13-fcf8f44b794d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 30 21:49:15.313839 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:15.313807 6720 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-8grjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 30 21:49:15.314228 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:15.313871 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grjn" podUID="b3dfec03-d265-405d-bc13-fcf8f44b794d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 30 21:49:16.314522 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:16.314494 6720 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-8grjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Oct 30 21:49:16.314907 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:16.314541 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grjn" podUID="b3dfec03-d265-405d-bc13-fcf8f44b794d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Oct 30 21:49:17.750442 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:17.750408 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grjn" Oct 30 21:49:17.750965 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:17.750693 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grjn" Oct 30 21:49:17.751788 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:17.751769 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grjn" Oct 30 21:49:24.013968 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:24.013934 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gcxhh"] Oct 30 21:49:24.022584 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:24.022561 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gcxhh" Oct 30 21:49:24.032837 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:24.032813 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gcxhh"] Oct 30 21:49:24.101221 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:24.101184 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac5c00c4-2343-47bd-805d-adc1bdf138f6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gcxhh\" (UID: \"ac5c00c4-2343-47bd-805d-adc1bdf138f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gcxhh" Oct 30 21:49:24.101431 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:24.101240 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac5c00c4-2343-47bd-805d-adc1bdf138f6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gcxhh\" (UID: \"ac5c00c4-2343-47bd-805d-adc1bdf138f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gcxhh" Oct 30 21:49:24.101431 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:24.101331 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac5c00c4-2343-47bd-805d-adc1bdf138f6-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gcxhh\" (UID: \"ac5c00c4-2343-47bd-805d-adc1bdf138f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gcxhh" Oct 30 21:49:24.101431 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:24.101366 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac5c00c4-2343-47bd-805d-adc1bdf138f6-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gcxhh\" (UID: \"ac5c00c4-2343-47bd-805d-adc1bdf138f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gcxhh" Oct 30 21:49:24.101611 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:24.101459 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnn75\" (UniqueName: \"kubernetes.io/projected/ac5c00c4-2343-47bd-805d-adc1bdf138f6-kube-api-access-jnn75\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gcxhh\" (UID: \"ac5c00c4-2343-47bd-805d-adc1bdf138f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gcxhh" Oct 30 21:49:24.101611 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:24.101502 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac5c00c4-2343-47bd-805d-adc1bdf138f6-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gcxhh\" (UID: \"ac5c00c4-2343-47bd-805d-adc1bdf138f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gcxhh" Oct 30 21:49:24.101611 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:24.101595 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac5c00c4-2343-47bd-805d-adc1bdf138f6-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gcxhh\" (UID: \"ac5c00c4-2343-47bd-805d-adc1bdf138f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gcxhh" Oct 30 21:49:24.101722 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:24.101631 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac5c00c4-2343-47bd-805d-adc1bdf138f6-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gcxhh\" (UID: \"ac5c00c4-2343-47bd-805d-adc1bdf138f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gcxhh" Oct 30 21:49:24.101722 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:24.101650 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac5c00c4-2343-47bd-805d-adc1bdf138f6-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gcxhh\" (UID: \"ac5c00c4-2343-47bd-805d-adc1bdf138f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gcxhh" Oct 30 21:49:24.202422 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:24.202302 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac5c00c4-2343-47bd-805d-adc1bdf138f6-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gcxhh\" (UID: \"ac5c00c4-2343-47bd-805d-adc1bdf138f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gcxhh" Oct 30 21:49:24.202422 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:24.202356 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac5c00c4-2343-47bd-805d-adc1bdf138f6-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gcxhh\" (UID: \"ac5c00c4-2343-47bd-805d-adc1bdf138f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gcxhh" Oct 30 21:49:24.202422 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:24.202418 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac5c00c4-2343-47bd-805d-adc1bdf138f6-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gcxhh\" (UID: \"ac5c00c4-2343-47bd-805d-adc1bdf138f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gcxhh" Oct 30 21:49:24.208815 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:24.202447 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac5c00c4-2343-47bd-805d-adc1bdf138f6-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gcxhh\" (UID: \"ac5c00c4-2343-47bd-805d-adc1bdf138f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gcxhh" Oct 30 21:49:24.208815 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:24.202510 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac5c00c4-2343-47bd-805d-adc1bdf138f6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gcxhh\" (UID: \"ac5c00c4-2343-47bd-805d-adc1bdf138f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gcxhh" Oct 30 21:49:24.208815 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:24.202560 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac5c00c4-2343-47bd-805d-adc1bdf138f6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gcxhh\" (UID: \"ac5c00c4-2343-47bd-805d-adc1bdf138f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gcxhh" Oct 30 21:49:24.208815 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:24.202646 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac5c00c4-2343-47bd-805d-adc1bdf138f6-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gcxhh\" (UID: \"ac5c00c4-2343-47bd-805d-adc1bdf138f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gcxhh" Oct 30 21:49:24.208815 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:24.202690 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac5c00c4-2343-47bd-805d-adc1bdf138f6-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gcxhh\" (UID: \"ac5c00c4-2343-47bd-805d-adc1bdf138f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gcxhh" Oct 30 21:49:24.208815 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:24.202745 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnn75\" (UniqueName: \"kubernetes.io/projected/ac5c00c4-2343-47bd-805d-adc1bdf138f6-kube-api-access-jnn75\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gcxhh\" (UID: \"ac5c00c4-2343-47bd-805d-adc1bdf138f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gcxhh" Oct 30 21:49:24.208815 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:24.202791 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac5c00c4-2343-47bd-805d-adc1bdf138f6-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gcxhh\" (UID: \"ac5c00c4-2343-47bd-805d-adc1bdf138f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gcxhh" Oct 30 21:49:24.208815 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:24.202915 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac5c00c4-2343-47bd-805d-adc1bdf138f6-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gcxhh\" (UID: \"ac5c00c4-2343-47bd-805d-adc1bdf138f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gcxhh" Oct 30 21:49:24.208815 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:24.202993 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac5c00c4-2343-47bd-805d-adc1bdf138f6-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gcxhh\" (UID: \"ac5c00c4-2343-47bd-805d-adc1bdf138f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gcxhh" Oct 30 21:49:24.208815 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:24.203113 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac5c00c4-2343-47bd-805d-adc1bdf138f6-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gcxhh\" (UID: \"ac5c00c4-2343-47bd-805d-adc1bdf138f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gcxhh" Oct 30 21:49:24.208815 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:24.203224 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac5c00c4-2343-47bd-805d-adc1bdf138f6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gcxhh\" (UID: \"ac5c00c4-2343-47bd-805d-adc1bdf138f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gcxhh" Oct 30 21:49:24.208815 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:24.204665 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac5c00c4-2343-47bd-805d-adc1bdf138f6-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gcxhh\" (UID: \"ac5c00c4-2343-47bd-805d-adc1bdf138f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gcxhh" Oct 30 21:49:24.208815 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:24.205058 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac5c00c4-2343-47bd-805d-adc1bdf138f6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gcxhh\" (UID: \"ac5c00c4-2343-47bd-805d-adc1bdf138f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gcxhh" Oct 30 21:49:24.215856 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:24.215829 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac5c00c4-2343-47bd-805d-adc1bdf138f6-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gcxhh\" (UID: \"ac5c00c4-2343-47bd-805d-adc1bdf138f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gcxhh" Oct 30 21:49:24.216001 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:24.215946 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnn75\" (UniqueName: \"kubernetes.io/projected/ac5c00c4-2343-47bd-805d-adc1bdf138f6-kube-api-access-jnn75\") pod \"maas-default-gateway-openshift-default-85544d4cd7-gcxhh\" (UID: \"ac5c00c4-2343-47bd-805d-adc1bdf138f6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gcxhh" Oct 30 21:49:24.336638 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:24.336575 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gcxhh" Oct 30 21:49:24.620271 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:24.618468 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gcxhh"] Oct 30 21:49:24.622373 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:49:24.622343 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac5c00c4_2343_47bd_805d_adc1bdf138f6.slice/crio-d7395ee923c5a4a3cf731f602337a3e595677322d86e14aa3909bf68f9fed24f WatchSource:0}: Error finding container d7395ee923c5a4a3cf731f602337a3e595677322d86e14aa3909bf68f9fed24f: Status 404 returned error can't find the container with id d7395ee923c5a4a3cf731f602337a3e595677322d86e14aa3909bf68f9fed24f Oct 30 21:49:24.624721 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:24.624694 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:49:24.624859 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:24.624754 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:49:24.624859 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:24.624785 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:49:25.329844 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:25.329809 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gcxhh" event={"ID":"ac5c00c4-2343-47bd-805d-adc1bdf138f6","Type":"ContainerStarted","Data":"5a8d3f6dabaf4a4a5c0c0c354fdcfa292bfb0a3e326b44ed03289be899c08f0b"} Oct 30 21:49:25.330246 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:25.329871 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gcxhh" event={"ID":"ac5c00c4-2343-47bd-805d-adc1bdf138f6","Type":"ContainerStarted","Data":"d7395ee923c5a4a3cf731f602337a3e595677322d86e14aa3909bf68f9fed24f"} Oct 30 21:49:25.337760 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:25.337736 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gcxhh" Oct 30 21:49:25.339196 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:25.339178 6720 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-gcxhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 30 21:49:25.339258 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:25.339239 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gcxhh" podUID="ac5c00c4-2343-47bd-805d-adc1bdf138f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 30 21:49:25.370822 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:25.370764 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gcxhh" podStartSLOduration=2.370747918 podStartE2EDuration="2.370747918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:25.370054556 +0000 UTC m=+531.541019763" watchObservedRunningTime="2026-10-30 21:49:25.370747918 +0000 UTC m=+531.541713093" Oct 30 21:49:25.926951 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:25.926914 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:26.337962 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:26.337884 6720 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-gcxhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 30 21:49:26.337962 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:26.337943 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gcxhh" podUID="ac5c00c4-2343-47bd-805d-adc1bdf138f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 30 21:49:27.337083 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:27.337048 6720 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-gcxhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 30 21:49:27.337306 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:27.337122 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gcxhh" podUID="ac5c00c4-2343-47bd-805d-adc1bdf138f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 30 21:49:28.337480 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:28.337446 6720 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-gcxhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 30 21:49:28.337886 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:28.337502 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gcxhh" podUID="ac5c00c4-2343-47bd-805d-adc1bdf138f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 30 21:49:28.733356 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:28.733324 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:28.890435 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:28.890395 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:29.070031 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:29.069941 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:29.337858 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:29.337772 6720 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-gcxhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 30 21:49:29.337858 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:29.337842 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gcxhh" podUID="ac5c00c4-2343-47bd-805d-adc1bdf138f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 30 21:49:29.359995 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:29.359961 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:29.538405 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:29.538363 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:29.695285 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:29.695247 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:29.855077 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:29.855044 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:30.007292 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:30.007205 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:30.337542 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:30.337446 6720 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-gcxhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 30 21:49:30.337542 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:30.337511 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gcxhh" podUID="ac5c00c4-2343-47bd-805d-adc1bdf138f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 30 21:49:30.870638 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:30.870606 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:31.337291 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:31.337199 6720 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-gcxhh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Oct 30 21:49:31.337291 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:31.337263 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gcxhh" podUID="ac5c00c4-2343-47bd-805d-adc1bdf138f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Oct 30 21:49:31.342133 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:31.342106 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:31.589557 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:31.589457 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:31.914499 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:31.914460 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:32.070271 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:32.070241 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:32.221944 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:32.221847 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:32.847851 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:32.847800 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:32.988645 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:32.988601 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:33.113619 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:33.113592 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gcxhh" Oct 30 21:49:33.113921 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:33.113898 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gcxhh" Oct 30 21:49:33.114865 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:33.114848 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gcxhh" Oct 30 21:49:33.135365 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:33.135341 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:33.293892 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:33.293855 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:34.017510 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:34.017483 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:34.164419 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:34.164371 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:34.424056 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:34.424018 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:34.657665 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:34.657629 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:34.836286 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:34.836148 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:35.848786 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:35.848735 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:36.017127 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:36.017089 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:36.643402 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:36.643346 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:37.332214 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:37.332180 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-367ad753-568c785748-7d9t8"] Oct 30 21:49:37.337696 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:37.337672 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-367ad753-568c785748-7d9t8" Oct 30 21:49:37.347945 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:37.347924 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-367ad753-serving-cert\"" Oct 30 21:49:37.349576 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:37.349556 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 21:49:37.349576 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:37.349559 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rm8tr\"" Oct 30 21:49:37.359009 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:37.358984 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-367ad753-568c785748-7d9t8"] Oct 30 21:49:37.451917 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:37.451874 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:37.464174 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:37.464141 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2d80d20-fca2-4e39-aa11-13b2790bf4c8-metrics-certs\") pod \"maas-api-e2e-ait-367ad753-568c785748-7d9t8\" (UID: \"f2d80d20-fca2-4e39-aa11-13b2790bf4c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-367ad753-568c785748-7d9t8" Oct 30 21:49:37.464336 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:37.464190 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2d80d20-fca2-4e39-aa11-13b2790bf4c8-maas-api-tls\") pod \"maas-api-e2e-ait-367ad753-568c785748-7d9t8\" (UID: \"f2d80d20-fca2-4e39-aa11-13b2790bf4c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-367ad753-568c785748-7d9t8" Oct 30 21:49:37.464336 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:37.464256 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2d80d20-fca2-4e39-aa11-13b2790bf4c8-openshift-service-ca\") pod \"maas-api-e2e-ait-367ad753-568c785748-7d9t8\" (UID: \"f2d80d20-fca2-4e39-aa11-13b2790bf4c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-367ad753-568c785748-7d9t8" Oct 30 21:49:37.464336 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:37.464317 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85v57\" (UniqueName: \"kubernetes.io/projected/f2d80d20-fca2-4e39-aa11-13b2790bf4c8-kube-api-access-85v57\") pod \"maas-api-e2e-ait-367ad753-568c785748-7d9t8\" (UID: \"f2d80d20-fca2-4e39-aa11-13b2790bf4c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-367ad753-568c785748-7d9t8" Oct 30 21:49:37.565363 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:37.565335 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85v57\" (UniqueName: \"kubernetes.io/projected/f2d80d20-fca2-4e39-aa11-13b2790bf4c8-kube-api-access-85v57\") pod \"maas-api-e2e-ait-367ad753-568c785748-7d9t8\" (UID: \"f2d80d20-fca2-4e39-aa11-13b2790bf4c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-367ad753-568c785748-7d9t8" Oct 30 21:49:37.565513 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:37.565420 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2d80d20-fca2-4e39-aa11-13b2790bf4c8-metrics-certs\") pod \"maas-api-e2e-ait-367ad753-568c785748-7d9t8\" (UID: \"f2d80d20-fca2-4e39-aa11-13b2790bf4c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-367ad753-568c785748-7d9t8" Oct 30 21:49:37.565513 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:37.565450 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2d80d20-fca2-4e39-aa11-13b2790bf4c8-maas-api-tls\") pod \"maas-api-e2e-ait-367ad753-568c785748-7d9t8\" (UID: \"f2d80d20-fca2-4e39-aa11-13b2790bf4c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-367ad753-568c785748-7d9t8" Oct 30 21:49:37.565513 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:37.565488 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2d80d20-fca2-4e39-aa11-13b2790bf4c8-openshift-service-ca\") pod \"maas-api-e2e-ait-367ad753-568c785748-7d9t8\" (UID: \"f2d80d20-fca2-4e39-aa11-13b2790bf4c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-367ad753-568c785748-7d9t8" Oct 30 21:49:37.566267 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:37.566199 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2d80d20-fca2-4e39-aa11-13b2790bf4c8-openshift-service-ca\") pod \"maas-api-e2e-ait-367ad753-568c785748-7d9t8\" (UID: \"f2d80d20-fca2-4e39-aa11-13b2790bf4c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-367ad753-568c785748-7d9t8" Oct 30 21:49:37.567971 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:37.567947 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2d80d20-fca2-4e39-aa11-13b2790bf4c8-maas-api-tls\") pod \"maas-api-e2e-ait-367ad753-568c785748-7d9t8\" (UID: \"f2d80d20-fca2-4e39-aa11-13b2790bf4c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-367ad753-568c785748-7d9t8" Oct 30 21:49:37.568113 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:37.568095 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2d80d20-fca2-4e39-aa11-13b2790bf4c8-metrics-certs\") pod \"maas-api-e2e-ait-367ad753-568c785748-7d9t8\" (UID: \"f2d80d20-fca2-4e39-aa11-13b2790bf4c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-367ad753-568c785748-7d9t8" Oct 30 21:49:37.582379 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:37.582293 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85v57\" (UniqueName: \"kubernetes.io/projected/f2d80d20-fca2-4e39-aa11-13b2790bf4c8-kube-api-access-85v57\") pod \"maas-api-e2e-ait-367ad753-568c785748-7d9t8\" (UID: \"f2d80d20-fca2-4e39-aa11-13b2790bf4c8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-367ad753-568c785748-7d9t8" Oct 30 21:49:37.615631 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:37.615595 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:37.652706 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:37.652685 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-367ad753-568c785748-7d9t8" Oct 30 21:49:37.936163 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:37.936129 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-367ad753-568c785748-7d9t8"] Oct 30 21:49:37.943718 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:49:37.943691 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2d80d20_fca2_4e39_aa11_13b2790bf4c8.slice/crio-91a8e14b105d5d0783004373afe6ce810cf2d12a3c4437d1b13e7388541a056a WatchSource:0}: Error finding container 91a8e14b105d5d0783004373afe6ce810cf2d12a3c4437d1b13e7388541a056a: Status 404 returned error can't find the container with id 91a8e14b105d5d0783004373afe6ce810cf2d12a3c4437d1b13e7388541a056a Oct 30 21:49:38.377486 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:38.377441 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:38.409758 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:38.409724 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-367ad753-568c785748-7d9t8" event={"ID":"f2d80d20-fca2-4e39-aa11-13b2790bf4c8","Type":"ContainerStarted","Data":"91a8e14b105d5d0783004373afe6ce810cf2d12a3c4437d1b13e7388541a056a"} Oct 30 21:49:38.689025 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:38.688996 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:38.974289 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:38.974210 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:39.284290 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:39.284198 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:39.419422 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:39.419351 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-367ad753-568c785748-7d9t8" event={"ID":"f2d80d20-fca2-4e39-aa11-13b2790bf4c8","Type":"ContainerStarted","Data":"1e20754d2b41a4494e93332688da57c7b4d22f2b9a4679c823cc14d0c0dd004e"} Oct 30 21:49:39.419928 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:39.419557 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-367ad753-568c785748-7d9t8" Oct 30 21:49:39.445240 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:39.445207 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:39.445719 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:39.445668 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-367ad753-568c785748-7d9t8" podStartSLOduration=2.040052341 podStartE2EDuration="2.445648333s" totalImagesPullingTime="405.595992ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:39.444049417 +0000 UTC m=+545.615014600" watchObservedRunningTime="2026-10-30 21:49:39.445648333 +0000 UTC m=+545.616613519" Oct 30 21:49:39.888356 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:39.888310 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:40.402406 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:49:40.402345 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 21:49:40.529762 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:40.529732 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:41.334448 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:41.334416 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:42.118871 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:42.118836 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:42.354028 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:42.353994 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:43.024184 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:43.024147 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:43.385857 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:43.385759 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:43.426585 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:43.426555 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4487d16a-7fc777df57-fhvwz"] Oct 30 21:49:43.430366 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:43.430350 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-4487d16a-7fc777df57-fhvwz" Oct 30 21:49:43.435545 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:43.435523 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-4487d16a-dockercfg-9h99k\"" Oct 30 21:49:43.454609 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:43.454576 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4487d16a-7fc777df57-fhvwz"] Oct 30 21:49:43.515567 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:43.515537 6720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-4487d16a\"" Oct 30 21:49:43.519119 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:43.519084 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2aa65fa-e6bd-4130-ba31-136478aa89e3-plugins-config-volume\") pod \"payload-processing-e2e-derive-4487d16a-7fc777df57-fhvwz\" (UID: \"c2aa65fa-e6bd-4130-ba31-136478aa89e3\") " pod="openshift-ingress/payload-processing-e2e-derive-4487d16a-7fc777df57-fhvwz" Oct 30 21:49:43.519300 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:43.519152 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7fkt\" (UniqueName: \"kubernetes.io/projected/c2aa65fa-e6bd-4130-ba31-136478aa89e3-kube-api-access-w7fkt\") pod \"payload-processing-e2e-derive-4487d16a-7fc777df57-fhvwz\" (UID: \"c2aa65fa-e6bd-4130-ba31-136478aa89e3\") " pod="openshift-ingress/payload-processing-e2e-derive-4487d16a-7fc777df57-fhvwz" Oct 30 21:49:43.545634 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:43.545598 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:43.619817 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:43.619782 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2aa65fa-e6bd-4130-ba31-136478aa89e3-plugins-config-volume\") pod \"payload-processing-e2e-derive-4487d16a-7fc777df57-fhvwz\" (UID: \"c2aa65fa-e6bd-4130-ba31-136478aa89e3\") " pod="openshift-ingress/payload-processing-e2e-derive-4487d16a-7fc777df57-fhvwz" Oct 30 21:49:43.619990 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:43.619845 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7fkt\" (UniqueName: \"kubernetes.io/projected/c2aa65fa-e6bd-4130-ba31-136478aa89e3-kube-api-access-w7fkt\") pod \"payload-processing-e2e-derive-4487d16a-7fc777df57-fhvwz\" (UID: \"c2aa65fa-e6bd-4130-ba31-136478aa89e3\") " pod="openshift-ingress/payload-processing-e2e-derive-4487d16a-7fc777df57-fhvwz" Oct 30 21:49:43.620585 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:43.620566 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2aa65fa-e6bd-4130-ba31-136478aa89e3-plugins-config-volume\") pod \"payload-processing-e2e-derive-4487d16a-7fc777df57-fhvwz\" (UID: \"c2aa65fa-e6bd-4130-ba31-136478aa89e3\") " pod="openshift-ingress/payload-processing-e2e-derive-4487d16a-7fc777df57-fhvwz" Oct 30 21:49:43.633273 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:43.633246 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7fkt\" (UniqueName: \"kubernetes.io/projected/c2aa65fa-e6bd-4130-ba31-136478aa89e3-kube-api-access-w7fkt\") pod \"payload-processing-e2e-derive-4487d16a-7fc777df57-fhvwz\" (UID: \"c2aa65fa-e6bd-4130-ba31-136478aa89e3\") " pod="openshift-ingress/payload-processing-e2e-derive-4487d16a-7fc777df57-fhvwz" Oct 30 21:49:43.747231 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:43.747156 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-4487d16a-7fc777df57-fhvwz" Oct 30 21:49:44.026509 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:44.026432 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4487d16a-7fc777df57-fhvwz"] Oct 30 21:49:44.029299 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:49:44.029269 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2aa65fa_e6bd_4130_ba31_136478aa89e3.slice/crio-7949621f22e209e58fbd305be7b3207a1373d323046f9193f60f3bda1d934301 WatchSource:0}: Error finding container 7949621f22e209e58fbd305be7b3207a1373d323046f9193f60f3bda1d934301: Status 404 returned error can't find the container with id 7949621f22e209e58fbd305be7b3207a1373d323046f9193f60f3bda1d934301 Oct 30 21:49:44.159412 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:44.158528 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:44.445257 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:44.445220 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-4487d16a-7fc777df57-fhvwz" event={"ID":"c2aa65fa-e6bd-4130-ba31-136478aa89e3","Type":"ContainerStarted","Data":"7949621f22e209e58fbd305be7b3207a1373d323046f9193f60f3bda1d934301"} Oct 30 21:49:45.011926 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:45.011891 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:45.431166 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:45.431139 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-367ad753-568c785748-7d9t8" Oct 30 21:49:45.913302 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:45.913267 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:46.066025 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:46.065998 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:46.229315 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:46.229284 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:46.456922 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:46.456836 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-4487d16a-7fc777df57-fhvwz" event={"ID":"c2aa65fa-e6bd-4130-ba31-136478aa89e3","Type":"ContainerStarted","Data":"ea349c810dc34443aaf584f700d6de52a97357e553a89bbe2c50907360b7960f"} Oct 30 21:49:46.457069 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:46.456959 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-4487d16a-7fc777df57-fhvwz" Oct 30 21:49:46.519144 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:46.519091 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-4487d16a-7fc777df57-fhvwz" podStartSLOduration=1.4425373910000001 podStartE2EDuration="3.519075049s" totalImagesPullingTime="2.076537658s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:46.517319344 +0000 UTC m=+552.688284537" watchObservedRunningTime="2026-10-30 21:49:46.519075049 +0000 UTC m=+552.690040231" Oct 30 21:49:47.053182 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:47.053141 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:47.749786 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:47.749751 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:48.497228 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:48.497191 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:49.522889 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:49.522858 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:50.014214 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:50.014173 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9528ebfc-gv5tm"] Oct 30 21:49:50.018482 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:50.018457 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9528ebfc-gv5tm" Oct 30 21:49:50.023201 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:50.023178 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:49:50.023441 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:50.023423 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 21:49:50.033715 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:50.033685 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9528ebfc-gv5tm"] Oct 30 21:49:50.102459 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:50.102430 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/743b707d-1c58-45de-a4df-f569d1ea39a7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-9528ebfc-gv5tm\" (UID: \"743b707d-1c58-45de-a4df-f569d1ea39a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9528ebfc-gv5tm" Oct 30 21:49:50.102621 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:50.102491 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/743b707d-1c58-45de-a4df-f569d1ea39a7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-9528ebfc-gv5tm\" (UID: \"743b707d-1c58-45de-a4df-f569d1ea39a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9528ebfc-gv5tm" Oct 30 21:49:50.102621 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:50.102590 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64ddw\" (UniqueName: \"kubernetes.io/projected/743b707d-1c58-45de-a4df-f569d1ea39a7-kube-api-access-64ddw\") pod \"maas-api-revoke-keys-e2e-ait-9528ebfc-gv5tm\" (UID: \"743b707d-1c58-45de-a4df-f569d1ea39a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9528ebfc-gv5tm" Oct 30 21:49:50.203865 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:50.203824 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/743b707d-1c58-45de-a4df-f569d1ea39a7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-9528ebfc-gv5tm\" (UID: \"743b707d-1c58-45de-a4df-f569d1ea39a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9528ebfc-gv5tm" Oct 30 21:49:50.204044 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:50.203885 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/743b707d-1c58-45de-a4df-f569d1ea39a7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-9528ebfc-gv5tm\" (UID: \"743b707d-1c58-45de-a4df-f569d1ea39a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9528ebfc-gv5tm" Oct 30 21:49:50.204044 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:50.203922 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64ddw\" (UniqueName: \"kubernetes.io/projected/743b707d-1c58-45de-a4df-f569d1ea39a7-kube-api-access-64ddw\") pod \"maas-api-revoke-keys-e2e-ait-9528ebfc-gv5tm\" (UID: \"743b707d-1c58-45de-a4df-f569d1ea39a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9528ebfc-gv5tm" Oct 30 21:49:50.204567 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:50.204537 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/743b707d-1c58-45de-a4df-f569d1ea39a7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-9528ebfc-gv5tm\" (UID: \"743b707d-1c58-45de-a4df-f569d1ea39a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9528ebfc-gv5tm" Oct 30 21:49:50.206341 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:50.206319 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/743b707d-1c58-45de-a4df-f569d1ea39a7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-9528ebfc-gv5tm\" (UID: \"743b707d-1c58-45de-a4df-f569d1ea39a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9528ebfc-gv5tm" Oct 30 21:49:50.217061 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:50.217031 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64ddw\" (UniqueName: \"kubernetes.io/projected/743b707d-1c58-45de-a4df-f569d1ea39a7-kube-api-access-64ddw\") pod \"maas-api-revoke-keys-e2e-ait-9528ebfc-gv5tm\" (UID: \"743b707d-1c58-45de-a4df-f569d1ea39a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9528ebfc-gv5tm" Oct 30 21:49:50.331713 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:50.331632 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9528ebfc-gv5tm" Oct 30 21:49:50.597006 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:50.596920 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:50.638343 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:50.638316 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9528ebfc-gv5tm"] Oct 30 21:49:50.641558 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:49:50.641530 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod743b707d_1c58_45de_a4df_f569d1ea39a7.slice/crio-595cbddcf971ac427aa67f3537dcb2fe2ada56fbec6b4fd2e2945cc78d9b438f WatchSource:0}: Error finding container 595cbddcf971ac427aa67f3537dcb2fe2ada56fbec6b4fd2e2945cc78d9b438f: Status 404 returned error can't find the container with id 595cbddcf971ac427aa67f3537dcb2fe2ada56fbec6b4fd2e2945cc78d9b438f Oct 30 21:49:51.283141 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:51.283106 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:51.485342 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:51.485308 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9528ebfc-gv5tm" event={"ID":"743b707d-1c58-45de-a4df-f569d1ea39a7","Type":"ContainerStarted","Data":"595cbddcf971ac427aa67f3537dcb2fe2ada56fbec6b4fd2e2945cc78d9b438f"} Oct 30 21:49:51.939542 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:51.939502 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:52.618084 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:52.618049 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:53.417503 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:53.417473 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:54.225825 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:54.225795 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:54.502115 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:54.502080 6720 generic.go:356] "Generic (PLEG): container finished" podID="743b707d-1c58-45de-a4df-f569d1ea39a7" containerID="54006fa841d58f3cc02209ab1e349c816b4104e504b334518de4a990769390b8" exitCode=0 Oct 30 21:49:54.502570 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:54.502162 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9528ebfc-gv5tm" event={"ID":"743b707d-1c58-45de-a4df-f569d1ea39a7","Type":"ContainerDied","Data":"54006fa841d58f3cc02209ab1e349c816b4104e504b334518de4a990769390b8"} Oct 30 21:49:55.036490 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:55.036448 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:55.650022 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:55.649998 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9528ebfc-gv5tm" Oct 30 21:49:55.721024 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:55.720972 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:55.764711 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:55.764631 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/743b707d-1c58-45de-a4df-f569d1ea39a7-kube-api-access-64ddw\" (UniqueName: \"kubernetes.io/projected/743b707d-1c58-45de-a4df-f569d1ea39a7-kube-api-access-64ddw\") pod \"743b707d-1c58-45de-a4df-f569d1ea39a7\" (UID: \"743b707d-1c58-45de-a4df-f569d1ea39a7\") " Oct 30 21:49:55.764869 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:55.764775 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/743b707d-1c58-45de-a4df-f569d1ea39a7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/743b707d-1c58-45de-a4df-f569d1ea39a7-maas-api-openshift-ca\") pod \"743b707d-1c58-45de-a4df-f569d1ea39a7\" (UID: \"743b707d-1c58-45de-a4df-f569d1ea39a7\") " Oct 30 21:49:55.764945 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:55.764883 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/743b707d-1c58-45de-a4df-f569d1ea39a7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/743b707d-1c58-45de-a4df-f569d1ea39a7-maas-api-xks-ca\") pod \"743b707d-1c58-45de-a4df-f569d1ea39a7\" (UID: \"743b707d-1c58-45de-a4df-f569d1ea39a7\") " Oct 30 21:49:55.765147 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:55.765125 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/743b707d-1c58-45de-a4df-f569d1ea39a7-maas-api-openshift-ca" pod "743b707d-1c58-45de-a4df-f569d1ea39a7" (UID: "743b707d-1c58-45de-a4df-f569d1ea39a7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:49:55.766967 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:55.766943 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/743b707d-1c58-45de-a4df-f569d1ea39a7-maas-api-xks-ca" pod "743b707d-1c58-45de-a4df-f569d1ea39a7" (UID: "743b707d-1c58-45de-a4df-f569d1ea39a7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:49:55.767050 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:55.766991 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/743b707d-1c58-45de-a4df-f569d1ea39a7-kube-api-access-64ddw" pod "743b707d-1c58-45de-a4df-f569d1ea39a7" (UID: "743b707d-1c58-45de-a4df-f569d1ea39a7"). InnerVolumeSpecName "kube-api-access-64ddw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:55.866636 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:55.866606 6720 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/743b707d-1c58-45de-a4df-f569d1ea39a7-maas-api-openshift-ca\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:55.866636 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:55.866632 6720 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/743b707d-1c58-45de-a4df-f569d1ea39a7-maas-api-xks-ca\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:55.866636 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:55.866642 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64ddw\" (UniqueName: \"kubernetes.io/projected/743b707d-1c58-45de-a4df-f569d1ea39a7-kube-api-access-64ddw\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:56.513121 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:56.513093 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9528ebfc-gv5tm" Oct 30 21:49:56.513297 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:56.513092 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9528ebfc-gv5tm" event={"ID":"743b707d-1c58-45de-a4df-f569d1ea39a7","Type":"ContainerDied","Data":"595cbddcf971ac427aa67f3537dcb2fe2ada56fbec6b4fd2e2945cc78d9b438f"} Oct 30 21:49:56.513297 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:56.513212 6720 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="595cbddcf971ac427aa67f3537dcb2fe2ada56fbec6b4fd2e2945cc78d9b438f" Oct 30 21:49:56.666093 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:56.666057 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:56.834022 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:56.833948 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:56.933016 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:56.932984 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4487d16a-qrx2z"] Oct 30 21:49:56.933437 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:56.933422 6720 cpu_manager.go:414] "containerMap: removing container" podUID="743b707d-1c58-45de-a4df-f569d1ea39a7" containerName="revoke-keys" Oct 30 21:49:56.933495 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:56.933438 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="743b707d-1c58-45de-a4df-f569d1ea39a7" containerName="revoke-keys" Oct 30 21:49:56.933558 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:56.933548 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="743b707d-1c58-45de-a4df-f569d1ea39a7" containerName="revoke-keys" Oct 30 21:49:56.936983 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:56.936964 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4487d16a-qrx2z" Oct 30 21:49:56.947271 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:56.947236 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4487d16a-qrx2z"] Oct 30 21:49:56.949347 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:56.949259 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:49:56.949661 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:56.949640 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 21:49:57.078773 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:57.077642 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72pw9\" (UniqueName: \"kubernetes.io/projected/3fd1d515-42cf-44c9-baf6-8892c78a515f-kube-api-access-72pw9\") pod \"maas-api-revoke-keys-e2e-derive-4487d16a-qrx2z\" (UID: \"3fd1d515-42cf-44c9-baf6-8892c78a515f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4487d16a-qrx2z" Oct 30 21:49:57.078773 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:57.078181 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fd1d515-42cf-44c9-baf6-8892c78a515f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-4487d16a-qrx2z\" (UID: \"3fd1d515-42cf-44c9-baf6-8892c78a515f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4487d16a-qrx2z" Oct 30 21:49:57.078773 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:57.078360 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fd1d515-42cf-44c9-baf6-8892c78a515f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-4487d16a-qrx2z\" (UID: \"3fd1d515-42cf-44c9-baf6-8892c78a515f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4487d16a-qrx2z" Oct 30 21:49:57.179878 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:57.179840 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72pw9\" (UniqueName: \"kubernetes.io/projected/3fd1d515-42cf-44c9-baf6-8892c78a515f-kube-api-access-72pw9\") pod \"maas-api-revoke-keys-e2e-derive-4487d16a-qrx2z\" (UID: \"3fd1d515-42cf-44c9-baf6-8892c78a515f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4487d16a-qrx2z" Oct 30 21:49:57.180061 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:57.179900 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fd1d515-42cf-44c9-baf6-8892c78a515f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-4487d16a-qrx2z\" (UID: \"3fd1d515-42cf-44c9-baf6-8892c78a515f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4487d16a-qrx2z" Oct 30 21:49:57.180061 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:57.180018 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fd1d515-42cf-44c9-baf6-8892c78a515f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-4487d16a-qrx2z\" (UID: \"3fd1d515-42cf-44c9-baf6-8892c78a515f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4487d16a-qrx2z" Oct 30 21:49:57.180658 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:57.180633 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fd1d515-42cf-44c9-baf6-8892c78a515f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-4487d16a-qrx2z\" (UID: \"3fd1d515-42cf-44c9-baf6-8892c78a515f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4487d16a-qrx2z" Oct 30 21:49:57.182353 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:57.182327 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fd1d515-42cf-44c9-baf6-8892c78a515f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-4487d16a-qrx2z\" (UID: \"3fd1d515-42cf-44c9-baf6-8892c78a515f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4487d16a-qrx2z" Oct 30 21:49:57.193231 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:57.193208 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72pw9\" (UniqueName: \"kubernetes.io/projected/3fd1d515-42cf-44c9-baf6-8892c78a515f-kube-api-access-72pw9\") pod \"maas-api-revoke-keys-e2e-derive-4487d16a-qrx2z\" (UID: \"3fd1d515-42cf-44c9-baf6-8892c78a515f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4487d16a-qrx2z" Oct 30 21:49:57.251067 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:57.251043 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4487d16a-qrx2z" Oct 30 21:49:57.465904 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:57.465764 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-4487d16a-7fc777df57-fhvwz" Oct 30 21:49:57.532341 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:57.532318 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4487d16a-qrx2z"] Oct 30 21:49:57.534986 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:49:57.534945 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fd1d515_42cf_44c9_baf6_8892c78a515f.slice/crio-2460dd07c7d6d61a51895fd7f038c1974113ab675f57d5ba382b52d24c5f784a WatchSource:0}: Error finding container 2460dd07c7d6d61a51895fd7f038c1974113ab675f57d5ba382b52d24c5f784a: Status 404 returned error can't find the container with id 2460dd07c7d6d61a51895fd7f038c1974113ab675f57d5ba382b52d24c5f784a Oct 30 21:49:57.587378 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:57.587356 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:58.506111 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:58.506080 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:58.525095 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:58.525062 6720 generic.go:356] "Generic (PLEG): container finished" podID="3fd1d515-42cf-44c9-baf6-8892c78a515f" containerID="e83e55aa77001994ca3eef57ce8b4a6fda551823c925ae0e576b125de4faebee" exitCode=0 Oct 30 21:49:58.525261 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:58.525125 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4487d16a-qrx2z" event={"ID":"3fd1d515-42cf-44c9-baf6-8892c78a515f","Type":"ContainerDied","Data":"e83e55aa77001994ca3eef57ce8b4a6fda551823c925ae0e576b125de4faebee"} Oct 30 21:49:58.525261 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:58.525168 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4487d16a-qrx2z" event={"ID":"3fd1d515-42cf-44c9-baf6-8892c78a515f","Type":"ContainerStarted","Data":"2460dd07c7d6d61a51895fd7f038c1974113ab675f57d5ba382b52d24c5f784a"} Oct 30 21:49:59.333130 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:59.333094 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:49:59.665766 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:59.665741 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4487d16a-qrx2z" Oct 30 21:49:59.805176 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:59.805149 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3fd1d515-42cf-44c9-baf6-8892c78a515f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fd1d515-42cf-44c9-baf6-8892c78a515f-maas-api-xks-ca\") pod \"3fd1d515-42cf-44c9-baf6-8892c78a515f\" (UID: \"3fd1d515-42cf-44c9-baf6-8892c78a515f\") " Oct 30 21:49:59.805350 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:59.805275 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fd1d515-42cf-44c9-baf6-8892c78a515f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fd1d515-42cf-44c9-baf6-8892c78a515f-maas-api-openshift-ca\") pod \"3fd1d515-42cf-44c9-baf6-8892c78a515f\" (UID: \"3fd1d515-42cf-44c9-baf6-8892c78a515f\") " Oct 30 21:49:59.805350 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:59.805320 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fd1d515-42cf-44c9-baf6-8892c78a515f-kube-api-access-72pw9\" (UniqueName: \"kubernetes.io/projected/3fd1d515-42cf-44c9-baf6-8892c78a515f-kube-api-access-72pw9\") pod \"3fd1d515-42cf-44c9-baf6-8892c78a515f\" (UID: \"3fd1d515-42cf-44c9-baf6-8892c78a515f\") " Oct 30 21:49:59.805674 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:59.805648 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fd1d515-42cf-44c9-baf6-8892c78a515f-maas-api-openshift-ca" pod "3fd1d515-42cf-44c9-baf6-8892c78a515f" (UID: "3fd1d515-42cf-44c9-baf6-8892c78a515f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:49:59.807397 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:59.807355 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fd1d515-42cf-44c9-baf6-8892c78a515f-maas-api-xks-ca" pod "3fd1d515-42cf-44c9-baf6-8892c78a515f" (UID: "3fd1d515-42cf-44c9-baf6-8892c78a515f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:49:59.807487 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:59.807464 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fd1d515-42cf-44c9-baf6-8892c78a515f-kube-api-access-72pw9" pod "3fd1d515-42cf-44c9-baf6-8892c78a515f" (UID: "3fd1d515-42cf-44c9-baf6-8892c78a515f"). InnerVolumeSpecName "kube-api-access-72pw9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:59.906882 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:59.906779 6720 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fd1d515-42cf-44c9-baf6-8892c78a515f-maas-api-openshift-ca\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:59.906882 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:59.906815 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72pw9\" (UniqueName: \"kubernetes.io/projected/3fd1d515-42cf-44c9-baf6-8892c78a515f-kube-api-access-72pw9\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:59.906882 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:59.906835 6720 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fd1d515-42cf-44c9-baf6-8892c78a515f-maas-api-xks-ca\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:49:59.995592 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:49:59.995560 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:00.279952 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:00.279880 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:00.428599 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:00.428561 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:00.536362 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:00.536283 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4487d16a-qrx2z" event={"ID":"3fd1d515-42cf-44c9-baf6-8892c78a515f","Type":"ContainerDied","Data":"2460dd07c7d6d61a51895fd7f038c1974113ab675f57d5ba382b52d24c5f784a"} Oct 30 21:50:00.536362 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:00.536304 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4487d16a-qrx2z" Oct 30 21:50:00.536362 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:00.536333 6720 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2460dd07c7d6d61a51895fd7f038c1974113ab675f57d5ba382b52d24c5f784a" Oct 30 21:50:00.646112 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:00.646081 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:01.322308 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:01.322259 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:02.122682 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:02.122646 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:02.821708 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:02.821673 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:02.971648 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:02.971613 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:03.327991 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:03.327958 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:03.472897 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:03.472861 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:03.626177 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:03.626136 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:04.080516 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:04.080435 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:04.211070 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:04.211041 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9528ebfc-gv5tm"] Oct 30 21:50:04.214783 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:04.214761 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-9528ebfc-gv5tm"] Oct 30 21:50:04.237149 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:04.237112 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:04.381994 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:04.381955 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:04.525440 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:04.525407 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:05.096119 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:05.096079 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:05.150038 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:05.150009 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-367ad753-568c785748-7d9t8"] Oct 30 21:50:05.150311 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:05.150258 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-367ad753-568c785748-7d9t8" podUID="f2d80d20-fca2-4e39-aa11-13b2790bf4c8" containerName="maas-api" containerID="cri-o://1e20754d2b41a4494e93332688da57c7b4d22f2b9a4679c823cc14d0c0dd004e" gracePeriod=30 Oct 30 21:50:05.246374 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:05.246340 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:05.406836 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:05.406697 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:05.425415 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:05.425361 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-367ad753-568c785748-7d9t8" Oct 30 21:50:05.556870 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:05.556830 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:05.563289 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:05.563256 6720 generic.go:356] "Generic (PLEG): container finished" podID="f2d80d20-fca2-4e39-aa11-13b2790bf4c8" containerID="1e20754d2b41a4494e93332688da57c7b4d22f2b9a4679c823cc14d0c0dd004e" exitCode=0 Oct 30 21:50:05.563497 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:05.563308 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-367ad753-568c785748-7d9t8" Oct 30 21:50:05.563497 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:05.563316 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-367ad753-568c785748-7d9t8" event={"ID":"f2d80d20-fca2-4e39-aa11-13b2790bf4c8","Type":"ContainerDied","Data":"1e20754d2b41a4494e93332688da57c7b4d22f2b9a4679c823cc14d0c0dd004e"} Oct 30 21:50:05.563497 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:05.563369 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-367ad753-568c785748-7d9t8" event={"ID":"f2d80d20-fca2-4e39-aa11-13b2790bf4c8","Type":"ContainerDied","Data":"91a8e14b105d5d0783004373afe6ce810cf2d12a3c4437d1b13e7388541a056a"} Oct 30 21:50:05.563497 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:05.563435 6720 scope.go:122] "RemoveContainer" containerID="1e20754d2b41a4494e93332688da57c7b4d22f2b9a4679c823cc14d0c0dd004e" Oct 30 21:50:05.569904 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:05.569051 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2d80d20-fca2-4e39-aa11-13b2790bf4c8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2d80d20-fca2-4e39-aa11-13b2790bf4c8-maas-api-tls\") pod \"f2d80d20-fca2-4e39-aa11-13b2790bf4c8\" (UID: \"f2d80d20-fca2-4e39-aa11-13b2790bf4c8\") " Oct 30 21:50:05.569904 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:05.569098 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2d80d20-fca2-4e39-aa11-13b2790bf4c8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2d80d20-fca2-4e39-aa11-13b2790bf4c8-metrics-certs\") pod \"f2d80d20-fca2-4e39-aa11-13b2790bf4c8\" (UID: \"f2d80d20-fca2-4e39-aa11-13b2790bf4c8\") " Oct 30 21:50:05.569904 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:05.569143 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2d80d20-fca2-4e39-aa11-13b2790bf4c8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2d80d20-fca2-4e39-aa11-13b2790bf4c8-openshift-service-ca\") pod \"f2d80d20-fca2-4e39-aa11-13b2790bf4c8\" (UID: \"f2d80d20-fca2-4e39-aa11-13b2790bf4c8\") " Oct 30 21:50:05.569904 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:05.569265 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2d80d20-fca2-4e39-aa11-13b2790bf4c8-kube-api-access-85v57\" (UniqueName: \"kubernetes.io/projected/f2d80d20-fca2-4e39-aa11-13b2790bf4c8-kube-api-access-85v57\") pod \"f2d80d20-fca2-4e39-aa11-13b2790bf4c8\" (UID: \"f2d80d20-fca2-4e39-aa11-13b2790bf4c8\") " Oct 30 21:50:05.569904 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:05.569607 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2d80d20-fca2-4e39-aa11-13b2790bf4c8-openshift-service-ca" pod "f2d80d20-fca2-4e39-aa11-13b2790bf4c8" (UID: "f2d80d20-fca2-4e39-aa11-13b2790bf4c8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:05.572160 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:05.572133 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2d80d20-fca2-4e39-aa11-13b2790bf4c8-maas-api-tls" pod "f2d80d20-fca2-4e39-aa11-13b2790bf4c8" (UID: "f2d80d20-fca2-4e39-aa11-13b2790bf4c8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:50:05.572464 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:05.572443 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2d80d20-fca2-4e39-aa11-13b2790bf4c8-metrics-certs" pod "f2d80d20-fca2-4e39-aa11-13b2790bf4c8" (UID: "f2d80d20-fca2-4e39-aa11-13b2790bf4c8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:50:05.572972 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:05.572948 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2d80d20-fca2-4e39-aa11-13b2790bf4c8-kube-api-access-85v57" pod "f2d80d20-fca2-4e39-aa11-13b2790bf4c8" (UID: "f2d80d20-fca2-4e39-aa11-13b2790bf4c8"). InnerVolumeSpecName "kube-api-access-85v57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:05.575580 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:05.575554 6720 scope.go:122] "RemoveContainer" containerID="1e20754d2b41a4494e93332688da57c7b4d22f2b9a4679c823cc14d0c0dd004e" Oct 30 21:50:05.575938 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:50:05.575914 6720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e20754d2b41a4494e93332688da57c7b4d22f2b9a4679c823cc14d0c0dd004e\": container with ID starting with 1e20754d2b41a4494e93332688da57c7b4d22f2b9a4679c823cc14d0c0dd004e not found: ID does not exist" containerID="1e20754d2b41a4494e93332688da57c7b4d22f2b9a4679c823cc14d0c0dd004e" Oct 30 21:50:05.576031 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:05.575953 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e20754d2b41a4494e93332688da57c7b4d22f2b9a4679c823cc14d0c0dd004e"} err="failed to get container status \"1e20754d2b41a4494e93332688da57c7b4d22f2b9a4679c823cc14d0c0dd004e\": rpc error: code = NotFound desc = could not find container \"1e20754d2b41a4494e93332688da57c7b4d22f2b9a4679c823cc14d0c0dd004e\": container with ID starting with 1e20754d2b41a4494e93332688da57c7b4d22f2b9a4679c823cc14d0c0dd004e not found: ID does not exist" Oct 30 21:50:05.671070 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:05.670998 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85v57\" (UniqueName: \"kubernetes.io/projected/f2d80d20-fca2-4e39-aa11-13b2790bf4c8-kube-api-access-85v57\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:05.671070 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:05.671029 6720 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2d80d20-fca2-4e39-aa11-13b2790bf4c8-maas-api-tls\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:05.671070 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:05.671040 6720 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2d80d20-fca2-4e39-aa11-13b2790bf4c8-metrics-certs\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:05.671070 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:05.671048 6720 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2d80d20-fca2-4e39-aa11-13b2790bf4c8-openshift-service-ca\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:05.728763 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:05.728734 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:05.884992 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:05.884940 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:05.891198 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:05.891161 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-367ad753-568c785748-7d9t8"] Oct 30 21:50:05.899406 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:05.899356 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-367ad753-568c785748-7d9t8"] Oct 30 21:50:05.956681 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:05.956610 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="743b707d-1c58-45de-a4df-f569d1ea39a7" path="/var/lib/kubelet/pods/743b707d-1c58-45de-a4df-f569d1ea39a7/volumes" Oct 30 21:50:05.957176 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:05.957157 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2d80d20-fca2-4e39-aa11-13b2790bf4c8" path="/var/lib/kubelet/pods/f2d80d20-fca2-4e39-aa11-13b2790bf4c8/volumes" Oct 30 21:50:06.091660 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:06.091628 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:06.465033 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:06.464994 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:06.610669 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:06.610631 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:06.778160 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:06.778071 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:06.944082 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:06.944048 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:07.170312 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:07.170271 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:07.249819 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:07.249787 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4487d16a-7fc777df57-fhvwz"] Oct 30 21:50:07.369269 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:07.369212 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:07.549470 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:07.546660 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:07.698663 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:07.698632 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:07.846960 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:07.846872 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:08.416550 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:08.416523 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:08.578296 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:08.578241 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:08.683728 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:08.683654 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4487d16a-qrx2z"] Oct 30 21:50:08.687188 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:08.687164 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-4487d16a-qrx2z"] Oct 30 21:50:08.726657 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:08.726629 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:08.977620 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:08.977546 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:09.147950 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:09.147918 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:09.297424 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:09.297310 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:09.882403 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:09.882361 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:09.957188 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:09.957141 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fd1d515-42cf-44c9-baf6-8892c78a515f" path="/var/lib/kubelet/pods/3fd1d515-42cf-44c9-baf6-8892c78a515f/volumes" Oct 30 21:50:10.039616 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:10.039589 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:10.391765 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:10.391678 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:10.413063 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:50:10.413025 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 21:50:10.569849 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:10.569808 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:11.030462 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:11.030429 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:11.276572 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:11.276536 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:11.612050 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:11.612010 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:11.798445 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:11.798410 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:12.026346 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:12.026269 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:12.222337 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:12.222302 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-ceb36a-7b9fbcd9d5-kssjr"] Oct 30 21:50:12.222829 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:12.222781 6720 cpu_manager.go:414] "containerMap: removing container" podUID="3fd1d515-42cf-44c9-baf6-8892c78a515f" containerName="revoke-keys" Oct 30 21:50:12.222829 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:12.222803 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fd1d515-42cf-44c9-baf6-8892c78a515f" containerName="revoke-keys" Oct 30 21:50:12.222940 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:12.222841 6720 cpu_manager.go:414] "containerMap: removing container" podUID="f2d80d20-fca2-4e39-aa11-13b2790bf4c8" containerName="maas-api" Oct 30 21:50:12.222940 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:12.222849 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2d80d20-fca2-4e39-aa11-13b2790bf4c8" containerName="maas-api" Oct 30 21:50:12.223047 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:12.222971 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fd1d515-42cf-44c9-baf6-8892c78a515f" containerName="revoke-keys" Oct 30 21:50:12.223047 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:12.222984 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2d80d20-fca2-4e39-aa11-13b2790bf4c8" containerName="maas-api" Oct 30 21:50:12.227749 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:12.227724 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ceb36a-7b9fbcd9d5-kssjr" Oct 30 21:50:12.241110 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:12.241091 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-ceb36a-serving-cert\"" Oct 30 21:50:12.241663 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:12.241648 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 21:50:12.241714 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:12.241648 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rm8tr\"" Oct 30 21:50:12.244774 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:12.244753 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-ceb36a-7b9fbcd9d5-kssjr"] Oct 30 21:50:12.250309 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:12.250265 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-4487d16a-7fc777df57-fhvwz" podUID="c2aa65fa-e6bd-4130-ba31-136478aa89e3" containerName="payload-processing" containerID="cri-o://ea349c810dc34443aaf584f700d6de52a97357e553a89bbe2c50907360b7960f" gracePeriod=25 Oct 30 21:50:12.335680 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:12.335589 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22c4c993-7bb7-492e-a4bc-05e7fa6c7b7f-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-ceb36a-7b9fbcd9d5-kssjr\" (UID: \"22c4c993-7bb7-492e-a4bc-05e7fa6c7b7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ceb36a-7b9fbcd9d5-kssjr" Oct 30 21:50:12.335680 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:12.335656 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22c4c993-7bb7-492e-a4bc-05e7fa6c7b7f-maas-api-tls\") pod \"maas-api-e2e-worker-w3-ceb36a-7b9fbcd9d5-kssjr\" (UID: \"22c4c993-7bb7-492e-a4bc-05e7fa6c7b7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ceb36a-7b9fbcd9d5-kssjr" Oct 30 21:50:12.335902 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:12.335758 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj2l6\" (UniqueName: \"kubernetes.io/projected/22c4c993-7bb7-492e-a4bc-05e7fa6c7b7f-kube-api-access-dj2l6\") pod \"maas-api-e2e-worker-w3-ceb36a-7b9fbcd9d5-kssjr\" (UID: \"22c4c993-7bb7-492e-a4bc-05e7fa6c7b7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ceb36a-7b9fbcd9d5-kssjr" Oct 30 21:50:12.335902 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:12.335875 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22c4c993-7bb7-492e-a4bc-05e7fa6c7b7f-metrics-certs\") pod \"maas-api-e2e-worker-w3-ceb36a-7b9fbcd9d5-kssjr\" (UID: \"22c4c993-7bb7-492e-a4bc-05e7fa6c7b7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ceb36a-7b9fbcd9d5-kssjr" Oct 30 21:50:12.339162 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:12.339129 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:12.437146 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:12.437111 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22c4c993-7bb7-492e-a4bc-05e7fa6c7b7f-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-ceb36a-7b9fbcd9d5-kssjr\" (UID: \"22c4c993-7bb7-492e-a4bc-05e7fa6c7b7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ceb36a-7b9fbcd9d5-kssjr" Oct 30 21:50:12.437321 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:12.437202 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22c4c993-7bb7-492e-a4bc-05e7fa6c7b7f-maas-api-tls\") pod \"maas-api-e2e-worker-w3-ceb36a-7b9fbcd9d5-kssjr\" (UID: \"22c4c993-7bb7-492e-a4bc-05e7fa6c7b7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ceb36a-7b9fbcd9d5-kssjr" Oct 30 21:50:12.437321 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:12.437232 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj2l6\" (UniqueName: \"kubernetes.io/projected/22c4c993-7bb7-492e-a4bc-05e7fa6c7b7f-kube-api-access-dj2l6\") pod \"maas-api-e2e-worker-w3-ceb36a-7b9fbcd9d5-kssjr\" (UID: \"22c4c993-7bb7-492e-a4bc-05e7fa6c7b7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ceb36a-7b9fbcd9d5-kssjr" Oct 30 21:50:12.437321 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:12.437316 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22c4c993-7bb7-492e-a4bc-05e7fa6c7b7f-metrics-certs\") pod \"maas-api-e2e-worker-w3-ceb36a-7b9fbcd9d5-kssjr\" (UID: \"22c4c993-7bb7-492e-a4bc-05e7fa6c7b7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ceb36a-7b9fbcd9d5-kssjr" Oct 30 21:50:12.437515 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:50:12.437345 6720 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-ceb36a-serving-cert: secret "maas-api-e2e-worker-w3-ceb36a-serving-cert" not found Oct 30 21:50:12.437515 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:50:12.437429 6720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22c4c993-7bb7-492e-a4bc-05e7fa6c7b7f-maas-api-tls podName:22c4c993-7bb7-492e-a4bc-05e7fa6c7b7f nodeName:}" failed. No retries permitted until 2026-10-30 21:50:12.937412696 +0000 UTC m=+579.108377877 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/22c4c993-7bb7-492e-a4bc-05e7fa6c7b7f-maas-api-tls") pod "maas-api-e2e-worker-w3-ceb36a-7b9fbcd9d5-kssjr" (UID: "22c4c993-7bb7-492e-a4bc-05e7fa6c7b7f") : secret "maas-api-e2e-worker-w3-ceb36a-serving-cert" not found Oct 30 21:50:12.437898 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:12.437859 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22c4c993-7bb7-492e-a4bc-05e7fa6c7b7f-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-ceb36a-7b9fbcd9d5-kssjr\" (UID: \"22c4c993-7bb7-492e-a4bc-05e7fa6c7b7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ceb36a-7b9fbcd9d5-kssjr" Oct 30 21:50:12.439952 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:12.439925 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22c4c993-7bb7-492e-a4bc-05e7fa6c7b7f-metrics-certs\") pod \"maas-api-e2e-worker-w3-ceb36a-7b9fbcd9d5-kssjr\" (UID: \"22c4c993-7bb7-492e-a4bc-05e7fa6c7b7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ceb36a-7b9fbcd9d5-kssjr" Oct 30 21:50:12.456272 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:12.456244 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj2l6\" (UniqueName: \"kubernetes.io/projected/22c4c993-7bb7-492e-a4bc-05e7fa6c7b7f-kube-api-access-dj2l6\") pod \"maas-api-e2e-worker-w3-ceb36a-7b9fbcd9d5-kssjr\" (UID: \"22c4c993-7bb7-492e-a4bc-05e7fa6c7b7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ceb36a-7b9fbcd9d5-kssjr" Oct 30 21:50:12.508365 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:12.508336 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:12.513558 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:12.513537 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-4487d16a-7fc777df57-fhvwz" Oct 30 21:50:12.603617 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:12.603584 6720 generic.go:356] "Generic (PLEG): container finished" podID="c2aa65fa-e6bd-4130-ba31-136478aa89e3" containerID="ea349c810dc34443aaf584f700d6de52a97357e553a89bbe2c50907360b7960f" exitCode=0 Oct 30 21:50:12.603797 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:12.603628 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-4487d16a-7fc777df57-fhvwz" event={"ID":"c2aa65fa-e6bd-4130-ba31-136478aa89e3","Type":"ContainerDied","Data":"ea349c810dc34443aaf584f700d6de52a97357e553a89bbe2c50907360b7960f"} Oct 30 21:50:12.603797 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:12.603637 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-4487d16a-7fc777df57-fhvwz" Oct 30 21:50:12.603797 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:12.603684 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-4487d16a-7fc777df57-fhvwz" event={"ID":"c2aa65fa-e6bd-4130-ba31-136478aa89e3","Type":"ContainerDied","Data":"7949621f22e209e58fbd305be7b3207a1373d323046f9193f60f3bda1d934301"} Oct 30 21:50:12.603797 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:12.603714 6720 scope.go:122] "RemoveContainer" containerID="ea349c810dc34443aaf584f700d6de52a97357e553a89bbe2c50907360b7960f" Oct 30 21:50:12.612803 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:12.612784 6720 scope.go:122] "RemoveContainer" containerID="ea349c810dc34443aaf584f700d6de52a97357e553a89bbe2c50907360b7960f" Oct 30 21:50:12.613092 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:50:12.613069 6720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea349c810dc34443aaf584f700d6de52a97357e553a89bbe2c50907360b7960f\": container with ID starting with ea349c810dc34443aaf584f700d6de52a97357e553a89bbe2c50907360b7960f not found: ID does not exist" containerID="ea349c810dc34443aaf584f700d6de52a97357e553a89bbe2c50907360b7960f" Oct 30 21:50:12.613191 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:12.613100 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea349c810dc34443aaf584f700d6de52a97357e553a89bbe2c50907360b7960f"} err="failed to get container status \"ea349c810dc34443aaf584f700d6de52a97357e553a89bbe2c50907360b7960f\": rpc error: code = NotFound desc = could not find container \"ea349c810dc34443aaf584f700d6de52a97357e553a89bbe2c50907360b7960f\": container with ID starting with ea349c810dc34443aaf584f700d6de52a97357e553a89bbe2c50907360b7960f not found: ID does not exist" Oct 30 21:50:12.639709 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:12.639683 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2aa65fa-e6bd-4130-ba31-136478aa89e3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2aa65fa-e6bd-4130-ba31-136478aa89e3-plugins-config-volume\") pod \"c2aa65fa-e6bd-4130-ba31-136478aa89e3\" (UID: \"c2aa65fa-e6bd-4130-ba31-136478aa89e3\") " Oct 30 21:50:12.639818 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:12.639720 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2aa65fa-e6bd-4130-ba31-136478aa89e3-kube-api-access-w7fkt\" (UniqueName: \"kubernetes.io/projected/c2aa65fa-e6bd-4130-ba31-136478aa89e3-kube-api-access-w7fkt\") pod \"c2aa65fa-e6bd-4130-ba31-136478aa89e3\" (UID: \"c2aa65fa-e6bd-4130-ba31-136478aa89e3\") " Oct 30 21:50:12.640023 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:12.640001 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2aa65fa-e6bd-4130-ba31-136478aa89e3-plugins-config-volume" pod "c2aa65fa-e6bd-4130-ba31-136478aa89e3" (UID: "c2aa65fa-e6bd-4130-ba31-136478aa89e3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:12.642093 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:12.642066 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2aa65fa-e6bd-4130-ba31-136478aa89e3-kube-api-access-w7fkt" pod "c2aa65fa-e6bd-4130-ba31-136478aa89e3" (UID: "c2aa65fa-e6bd-4130-ba31-136478aa89e3"). InnerVolumeSpecName "kube-api-access-w7fkt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:12.716471 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:12.716440 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:12.741308 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:12.741280 6720 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2aa65fa-e6bd-4130-ba31-136478aa89e3-plugins-config-volume\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:12.741308 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:12.741308 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7fkt\" (UniqueName: \"kubernetes.io/projected/c2aa65fa-e6bd-4130-ba31-136478aa89e3-kube-api-access-w7fkt\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:12.931322 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:12.931290 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:12.943346 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:12.943306 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22c4c993-7bb7-492e-a4bc-05e7fa6c7b7f-maas-api-tls\") pod \"maas-api-e2e-worker-w3-ceb36a-7b9fbcd9d5-kssjr\" (UID: \"22c4c993-7bb7-492e-a4bc-05e7fa6c7b7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ceb36a-7b9fbcd9d5-kssjr" Oct 30 21:50:12.946893 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:12.946865 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22c4c993-7bb7-492e-a4bc-05e7fa6c7b7f-maas-api-tls\") pod \"maas-api-e2e-worker-w3-ceb36a-7b9fbcd9d5-kssjr\" (UID: \"22c4c993-7bb7-492e-a4bc-05e7fa6c7b7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ceb36a-7b9fbcd9d5-kssjr" Oct 30 21:50:12.948739 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:12.948712 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4487d16a-7fc777df57-fhvwz"] Oct 30 21:50:12.958493 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:12.958466 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-4487d16a-7fc777df57-fhvwz"] Oct 30 21:50:13.126027 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:13.125996 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:13.141866 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:13.141841 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ceb36a-7b9fbcd9d5-kssjr" Oct 30 21:50:13.374920 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:13.374874 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:13.437993 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:13.437972 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-ceb36a-7b9fbcd9d5-kssjr"] Oct 30 21:50:13.441357 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:50:13.441324 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22c4c993_7bb7_492e_a4bc_05e7fa6c7b7f.slice/crio-34b151fbcd28be599c2bb2f838e96c06708c86aadcde66ff5778ad07b52155f0 WatchSource:0}: Error finding container 34b151fbcd28be599c2bb2f838e96c06708c86aadcde66ff5778ad07b52155f0: Status 404 returned error can't find the container with id 34b151fbcd28be599c2bb2f838e96c06708c86aadcde66ff5778ad07b52155f0 Oct 30 21:50:13.610794 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:13.610758 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ceb36a-7b9fbcd9d5-kssjr" event={"ID":"22c4c993-7bb7-492e-a4bc-05e7fa6c7b7f","Type":"ContainerStarted","Data":"34b151fbcd28be599c2bb2f838e96c06708c86aadcde66ff5778ad07b52155f0"} Oct 30 21:50:13.645115 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:13.645084 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:13.949452 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:13.949358 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:13.954340 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:13.954313 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2aa65fa-e6bd-4130-ba31-136478aa89e3" path="/var/lib/kubelet/pods/c2aa65fa-e6bd-4130-ba31-136478aa89e3/volumes" Oct 30 21:50:14.180215 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:14.180169 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:14.557659 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:14.557624 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:14.619186 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:14.619155 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ceb36a-7b9fbcd9d5-kssjr" event={"ID":"22c4c993-7bb7-492e-a4bc-05e7fa6c7b7f","Type":"ContainerStarted","Data":"60998a3d9c680ea9ddaa6eaf35d03a44b7e335890afcbc71657943bb7f92cba0"} Oct 30 21:50:14.619355 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:14.619305 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ceb36a-7b9fbcd9d5-kssjr" Oct 30 21:50:14.656893 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:14.656846 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ceb36a-7b9fbcd9d5-kssjr" podStartSLOduration=2.095344165 podStartE2EDuration="2.656830121s" totalImagesPullingTime="561.485956ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:14.655168525 +0000 UTC m=+580.826133708" watchObservedRunningTime="2026-10-30 21:50:14.656830121 +0000 UTC m=+580.827795302" Oct 30 21:50:14.754317 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:14.754276 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:15.307331 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:15.307299 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:15.509578 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:15.509543 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:15.790505 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:15.790467 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:15.947054 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:15.947024 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:16.251143 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:16.251114 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-1529dc-7d95c6c9d-9t9mq"] Oct 30 21:50:16.252041 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:16.252011 6720 cpu_manager.go:414] "containerMap: removing container" podUID="c2aa65fa-e6bd-4130-ba31-136478aa89e3" containerName="payload-processing" Oct 30 21:50:16.252041 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:16.252037 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2aa65fa-e6bd-4130-ba31-136478aa89e3" containerName="payload-processing" Oct 30 21:50:16.252227 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:16.252141 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2aa65fa-e6bd-4130-ba31-136478aa89e3" containerName="payload-processing" Oct 30 21:50:16.258572 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:16.258550 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-1529dc-7d95c6c9d-9t9mq"] Oct 30 21:50:16.258692 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:16.258638 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1529dc-7d95c6c9d-9t9mq" Oct 30 21:50:16.263113 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:16.263047 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-1529dc-serving-cert\"" Oct 30 21:50:16.272991 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:16.272966 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9cfa5a4-4d17-41f4-985c-c119257e3720-maas-api-tls\") pod \"maas-api-e2e-worker-w0-1529dc-7d95c6c9d-9t9mq\" (UID: \"c9cfa5a4-4d17-41f4-985c-c119257e3720\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1529dc-7d95c6c9d-9t9mq" Oct 30 21:50:16.273100 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:16.273008 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9cfa5a4-4d17-41f4-985c-c119257e3720-metrics-certs\") pod \"maas-api-e2e-worker-w0-1529dc-7d95c6c9d-9t9mq\" (UID: \"c9cfa5a4-4d17-41f4-985c-c119257e3720\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1529dc-7d95c6c9d-9t9mq" Oct 30 21:50:16.273100 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:16.273031 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfpf4\" (UniqueName: \"kubernetes.io/projected/c9cfa5a4-4d17-41f4-985c-c119257e3720-kube-api-access-vfpf4\") pod \"maas-api-e2e-worker-w0-1529dc-7d95c6c9d-9t9mq\" (UID: \"c9cfa5a4-4d17-41f4-985c-c119257e3720\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1529dc-7d95c6c9d-9t9mq" Oct 30 21:50:16.273188 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:16.273138 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9cfa5a4-4d17-41f4-985c-c119257e3720-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-1529dc-7d95c6c9d-9t9mq\" (UID: \"c9cfa5a4-4d17-41f4-985c-c119257e3720\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1529dc-7d95c6c9d-9t9mq" Oct 30 21:50:16.348278 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:16.348244 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:16.374108 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:16.374067 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9cfa5a4-4d17-41f4-985c-c119257e3720-maas-api-tls\") pod \"maas-api-e2e-worker-w0-1529dc-7d95c6c9d-9t9mq\" (UID: \"c9cfa5a4-4d17-41f4-985c-c119257e3720\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1529dc-7d95c6c9d-9t9mq" Oct 30 21:50:16.374108 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:16.374112 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9cfa5a4-4d17-41f4-985c-c119257e3720-metrics-certs\") pod \"maas-api-e2e-worker-w0-1529dc-7d95c6c9d-9t9mq\" (UID: \"c9cfa5a4-4d17-41f4-985c-c119257e3720\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1529dc-7d95c6c9d-9t9mq" Oct 30 21:50:16.374331 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:16.374133 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfpf4\" (UniqueName: \"kubernetes.io/projected/c9cfa5a4-4d17-41f4-985c-c119257e3720-kube-api-access-vfpf4\") pod \"maas-api-e2e-worker-w0-1529dc-7d95c6c9d-9t9mq\" (UID: \"c9cfa5a4-4d17-41f4-985c-c119257e3720\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1529dc-7d95c6c9d-9t9mq" Oct 30 21:50:16.374331 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:16.374152 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9cfa5a4-4d17-41f4-985c-c119257e3720-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-1529dc-7d95c6c9d-9t9mq\" (UID: \"c9cfa5a4-4d17-41f4-985c-c119257e3720\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1529dc-7d95c6c9d-9t9mq" Oct 30 21:50:16.374964 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:16.374856 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9cfa5a4-4d17-41f4-985c-c119257e3720-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-1529dc-7d95c6c9d-9t9mq\" (UID: \"c9cfa5a4-4d17-41f4-985c-c119257e3720\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1529dc-7d95c6c9d-9t9mq" Oct 30 21:50:16.376689 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:16.376663 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9cfa5a4-4d17-41f4-985c-c119257e3720-metrics-certs\") pod \"maas-api-e2e-worker-w0-1529dc-7d95c6c9d-9t9mq\" (UID: \"c9cfa5a4-4d17-41f4-985c-c119257e3720\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1529dc-7d95c6c9d-9t9mq" Oct 30 21:50:16.376857 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:16.376831 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9cfa5a4-4d17-41f4-985c-c119257e3720-maas-api-tls\") pod \"maas-api-e2e-worker-w0-1529dc-7d95c6c9d-9t9mq\" (UID: \"c9cfa5a4-4d17-41f4-985c-c119257e3720\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1529dc-7d95c6c9d-9t9mq" Oct 30 21:50:16.386862 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:16.386840 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfpf4\" (UniqueName: \"kubernetes.io/projected/c9cfa5a4-4d17-41f4-985c-c119257e3720-kube-api-access-vfpf4\") pod \"maas-api-e2e-worker-w0-1529dc-7d95c6c9d-9t9mq\" (UID: \"c9cfa5a4-4d17-41f4-985c-c119257e3720\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1529dc-7d95c6c9d-9t9mq" Oct 30 21:50:16.499417 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:16.499342 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:16.572849 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:16.572773 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1529dc-7d95c6c9d-9t9mq" Oct 30 21:50:16.852964 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:16.852914 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-1529dc-7d95c6c9d-9t9mq"] Oct 30 21:50:16.855171 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:50:16.855133 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9cfa5a4_4d17_41f4_985c_c119257e3720.slice/crio-e1d55f7857b0b02a074f5a20edfef3b36e0e918bf6de8f306b9749ffce8991eb WatchSource:0}: Error finding container e1d55f7857b0b02a074f5a20edfef3b36e0e918bf6de8f306b9749ffce8991eb: Status 404 returned error can't find the container with id e1d55f7857b0b02a074f5a20edfef3b36e0e918bf6de8f306b9749ffce8991eb Oct 30 21:50:17.131519 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:17.131484 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-939a16-86877564f-dvkvf"] Oct 30 21:50:17.141460 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:17.137725 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-939a16-86877564f-dvkvf" Oct 30 21:50:17.142239 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:17.142152 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-939a16-86877564f-dvkvf"] Oct 30 21:50:17.146088 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:17.146055 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-939a16-serving-cert\"" Oct 30 21:50:17.183500 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:17.183463 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f501cad-6ca4-4615-b978-bd05ee0cfa2c-maas-api-tls\") pod \"maas-api-e2e-worker-w4-939a16-86877564f-dvkvf\" (UID: \"2f501cad-6ca4-4615-b978-bd05ee0cfa2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-939a16-86877564f-dvkvf" Oct 30 21:50:17.183662 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:17.183638 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f501cad-6ca4-4615-b978-bd05ee0cfa2c-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-939a16-86877564f-dvkvf\" (UID: \"2f501cad-6ca4-4615-b978-bd05ee0cfa2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-939a16-86877564f-dvkvf" Oct 30 21:50:17.183750 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:17.183691 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f501cad-6ca4-4615-b978-bd05ee0cfa2c-metrics-certs\") pod \"maas-api-e2e-worker-w4-939a16-86877564f-dvkvf\" (UID: \"2f501cad-6ca4-4615-b978-bd05ee0cfa2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-939a16-86877564f-dvkvf" Oct 30 21:50:17.183820 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:17.183797 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spd9f\" (UniqueName: \"kubernetes.io/projected/2f501cad-6ca4-4615-b978-bd05ee0cfa2c-kube-api-access-spd9f\") pod \"maas-api-e2e-worker-w4-939a16-86877564f-dvkvf\" (UID: \"2f501cad-6ca4-4615-b978-bd05ee0cfa2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-939a16-86877564f-dvkvf" Oct 30 21:50:17.221225 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:17.221193 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:17.284915 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:17.284888 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f501cad-6ca4-4615-b978-bd05ee0cfa2c-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-939a16-86877564f-dvkvf\" (UID: \"2f501cad-6ca4-4615-b978-bd05ee0cfa2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-939a16-86877564f-dvkvf" Oct 30 21:50:17.285039 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:17.284936 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f501cad-6ca4-4615-b978-bd05ee0cfa2c-metrics-certs\") pod \"maas-api-e2e-worker-w4-939a16-86877564f-dvkvf\" (UID: \"2f501cad-6ca4-4615-b978-bd05ee0cfa2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-939a16-86877564f-dvkvf" Oct 30 21:50:17.285039 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:17.284967 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spd9f\" (UniqueName: \"kubernetes.io/projected/2f501cad-6ca4-4615-b978-bd05ee0cfa2c-kube-api-access-spd9f\") pod \"maas-api-e2e-worker-w4-939a16-86877564f-dvkvf\" (UID: \"2f501cad-6ca4-4615-b978-bd05ee0cfa2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-939a16-86877564f-dvkvf" Oct 30 21:50:17.285039 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:17.285029 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f501cad-6ca4-4615-b978-bd05ee0cfa2c-maas-api-tls\") pod \"maas-api-e2e-worker-w4-939a16-86877564f-dvkvf\" (UID: \"2f501cad-6ca4-4615-b978-bd05ee0cfa2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-939a16-86877564f-dvkvf" Oct 30 21:50:17.285647 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:17.285627 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f501cad-6ca4-4615-b978-bd05ee0cfa2c-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-939a16-86877564f-dvkvf\" (UID: \"2f501cad-6ca4-4615-b978-bd05ee0cfa2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-939a16-86877564f-dvkvf" Oct 30 21:50:17.288626 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:17.288596 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f501cad-6ca4-4615-b978-bd05ee0cfa2c-maas-api-tls\") pod \"maas-api-e2e-worker-w4-939a16-86877564f-dvkvf\" (UID: \"2f501cad-6ca4-4615-b978-bd05ee0cfa2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-939a16-86877564f-dvkvf" Oct 30 21:50:17.289208 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:17.288830 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f501cad-6ca4-4615-b978-bd05ee0cfa2c-metrics-certs\") pod \"maas-api-e2e-worker-w4-939a16-86877564f-dvkvf\" (UID: \"2f501cad-6ca4-4615-b978-bd05ee0cfa2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-939a16-86877564f-dvkvf" Oct 30 21:50:17.298071 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:17.298043 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spd9f\" (UniqueName: \"kubernetes.io/projected/2f501cad-6ca4-4615-b978-bd05ee0cfa2c-kube-api-access-spd9f\") pod \"maas-api-e2e-worker-w4-939a16-86877564f-dvkvf\" (UID: \"2f501cad-6ca4-4615-b978-bd05ee0cfa2c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-939a16-86877564f-dvkvf" Oct 30 21:50:17.411061 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:17.410971 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:17.488647 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:17.488602 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-939a16-86877564f-dvkvf" Oct 30 21:50:17.644666 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:17.644629 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1529dc-7d95c6c9d-9t9mq" event={"ID":"c9cfa5a4-4d17-41f4-985c-c119257e3720","Type":"ContainerStarted","Data":"d5afdc8d09368bf32f15a3c808da4a6f19bd77139c8c933648ec5d64eb8c84db"} Oct 30 21:50:17.644848 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:17.644683 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1529dc-7d95c6c9d-9t9mq" event={"ID":"c9cfa5a4-4d17-41f4-985c-c119257e3720","Type":"ContainerStarted","Data":"e1d55f7857b0b02a074f5a20edfef3b36e0e918bf6de8f306b9749ffce8991eb"} Oct 30 21:50:17.644848 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:17.644710 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1529dc-7d95c6c9d-9t9mq" Oct 30 21:50:17.668367 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:17.668273 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1529dc-7d95c6c9d-9t9mq" podStartSLOduration=1.259425569 podStartE2EDuration="1.668257466s" totalImagesPullingTime="408.831897ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:17.66650838 +0000 UTC m=+583.837473564" watchObservedRunningTime="2026-10-30 21:50:17.668257466 +0000 UTC m=+583.839222650" Oct 30 21:50:17.786253 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:17.786181 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-939a16-86877564f-dvkvf"] Oct 30 21:50:17.795425 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:50:17.792182 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f501cad_6ca4_4615_b978_bd05ee0cfa2c.slice/crio-31694c2dc677fd39d0d046571e86ae8d6367f0bb7200f843c69c37fbf7d8c35e WatchSource:0}: Error finding container 31694c2dc677fd39d0d046571e86ae8d6367f0bb7200f843c69c37fbf7d8c35e: Status 404 returned error can't find the container with id 31694c2dc677fd39d0d046571e86ae8d6367f0bb7200f843c69c37fbf7d8c35e Oct 30 21:50:17.927007 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:17.926905 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:18.600997 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:18.600922 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:18.652311 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:18.651951 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-939a16-86877564f-dvkvf" event={"ID":"2f501cad-6ca4-4615-b978-bd05ee0cfa2c","Type":"ContainerStarted","Data":"dfe42498929435064cb8601420efdad80c099517a0d209b74158bbc26b9dbdc0"} Oct 30 21:50:18.653242 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:18.652379 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-939a16-86877564f-dvkvf" event={"ID":"2f501cad-6ca4-4615-b978-bd05ee0cfa2c","Type":"ContainerStarted","Data":"31694c2dc677fd39d0d046571e86ae8d6367f0bb7200f843c69c37fbf7d8c35e"} Oct 30 21:50:18.653242 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:18.652434 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-939a16-86877564f-dvkvf" Oct 30 21:50:18.676571 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:18.676512 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-939a16-86877564f-dvkvf" podStartSLOduration=1.272507013 podStartE2EDuration="1.676497292s" totalImagesPullingTime="403.990279ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:18.675558602 +0000 UTC m=+584.846523785" watchObservedRunningTime="2026-10-30 21:50:18.676497292 +0000 UTC m=+584.847462475" Oct 30 21:50:18.742293 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:18.742263 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:19.349084 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:19.349051 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:20.248542 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:20.248508 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:20.634048 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:20.634022 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ceb36a-7b9fbcd9d5-kssjr" Oct 30 21:50:20.944546 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:20.944450 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:21.604500 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:21.604468 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:22.247086 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:22.247050 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:22.962341 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:22.962309 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:23.659268 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:23.659234 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1529dc-7d95c6c9d-9t9mq" Oct 30 21:50:23.894050 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:23.894009 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:24.642052 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:24.642017 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:24.663244 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:24.663219 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-939a16-86877564f-dvkvf" Oct 30 21:50:25.477218 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:25.477178 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:25.947141 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:25.947109 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:26.204354 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:26.204273 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:26.374023 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:26.373987 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:26.639086 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:26.639057 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:26.824839 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:26.824806 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:27.020050 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:27.019968 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:27.205537 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:27.205504 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:27.892225 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:27.892189 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:28.063082 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:28.063039 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:28.714027 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:28.713992 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:29.454999 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:29.454963 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:29.617110 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:29.617076 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:30.040131 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:30.040090 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:30.324378 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:30.324227 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:30.772940 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:30.772901 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:30.922057 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:30.922020 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:31.172739 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:31.172670 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:31.942555 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:31.942525 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:32.623903 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:32.623869 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:33.191986 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:33.191953 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:33.363967 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:33.363935 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:33.882113 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:33.882082 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:34.074961 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:34.074927 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:34.291207 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:34.291128 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:34.899935 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:34.899906 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:35.181183 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:35.181095 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:35.441355 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:35.441277 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:36.124690 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:36.124661 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:36.824518 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:36.824481 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:37.520582 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:37.520502 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:38.206650 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:38.206616 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-1529dc/simulator-w0-1529dc-kserve-8466d8d645-l54cf"] Oct 30 21:50:38.214994 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:38.214969 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1529dc/simulator-w0-1529dc-kserve-8466d8d645-l54cf" Oct 30 21:50:38.223890 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:38.223857 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:38.224377 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:38.224357 6720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-1529dc\"/\"openshift-service-ca.crt\"" Oct 30 21:50:38.225927 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:38.225905 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-1529dc\"/\"default-dockercfg-w5jph\"" Oct 30 21:50:38.226094 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:38.225930 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-1529dc\"/\"simulator-w0-1529dc-kserve-self-signed-certs\"" Oct 30 21:50:38.226173 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:38.225929 6720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-1529dc\"/\"kube-root-ca.crt\"" Oct 30 21:50:38.231842 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:38.231818 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1529dc/simulator-w0-1529dc-kserve-8466d8d645-l54cf"] Oct 30 21:50:38.294531 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:38.294491 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/257edb08-2e86-4f49-be57-848ace5958b1-home\") pod \"simulator-w0-1529dc-kserve-8466d8d645-l54cf\" (UID: \"257edb08-2e86-4f49-be57-848ace5958b1\") " pod="e2e-models-e2e-worker-w0-1529dc/simulator-w0-1529dc-kserve-8466d8d645-l54cf" Oct 30 21:50:38.294531 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:38.294542 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpglj\" (UniqueName: \"kubernetes.io/projected/257edb08-2e86-4f49-be57-848ace5958b1-kube-api-access-wpglj\") pod \"simulator-w0-1529dc-kserve-8466d8d645-l54cf\" (UID: \"257edb08-2e86-4f49-be57-848ace5958b1\") " pod="e2e-models-e2e-worker-w0-1529dc/simulator-w0-1529dc-kserve-8466d8d645-l54cf" Oct 30 21:50:38.294808 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:38.294641 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/257edb08-2e86-4f49-be57-848ace5958b1-tmp-dir\") pod \"simulator-w0-1529dc-kserve-8466d8d645-l54cf\" (UID: \"257edb08-2e86-4f49-be57-848ace5958b1\") " pod="e2e-models-e2e-worker-w0-1529dc/simulator-w0-1529dc-kserve-8466d8d645-l54cf" Oct 30 21:50:38.294808 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:38.294670 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/257edb08-2e86-4f49-be57-848ace5958b1-tls-certs\") pod \"simulator-w0-1529dc-kserve-8466d8d645-l54cf\" (UID: \"257edb08-2e86-4f49-be57-848ace5958b1\") " pod="e2e-models-e2e-worker-w0-1529dc/simulator-w0-1529dc-kserve-8466d8d645-l54cf" Oct 30 21:50:38.294926 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:38.294812 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/257edb08-2e86-4f49-be57-848ace5958b1-dshm\") pod \"simulator-w0-1529dc-kserve-8466d8d645-l54cf\" (UID: \"257edb08-2e86-4f49-be57-848ace5958b1\") " pod="e2e-models-e2e-worker-w0-1529dc/simulator-w0-1529dc-kserve-8466d8d645-l54cf" Oct 30 21:50:38.294926 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:38.294861 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/257edb08-2e86-4f49-be57-848ace5958b1-model-cache\") pod \"simulator-w0-1529dc-kserve-8466d8d645-l54cf\" (UID: \"257edb08-2e86-4f49-be57-848ace5958b1\") " pod="e2e-models-e2e-worker-w0-1529dc/simulator-w0-1529dc-kserve-8466d8d645-l54cf" Oct 30 21:50:38.389643 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:38.389610 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:38.396070 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:38.396045 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/257edb08-2e86-4f49-be57-848ace5958b1-dshm\") pod \"simulator-w0-1529dc-kserve-8466d8d645-l54cf\" (UID: \"257edb08-2e86-4f49-be57-848ace5958b1\") " pod="e2e-models-e2e-worker-w0-1529dc/simulator-w0-1529dc-kserve-8466d8d645-l54cf" Oct 30 21:50:38.396149 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:38.396086 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/257edb08-2e86-4f49-be57-848ace5958b1-model-cache\") pod \"simulator-w0-1529dc-kserve-8466d8d645-l54cf\" (UID: \"257edb08-2e86-4f49-be57-848ace5958b1\") " pod="e2e-models-e2e-worker-w0-1529dc/simulator-w0-1529dc-kserve-8466d8d645-l54cf" Oct 30 21:50:38.396149 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:38.396142 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/257edb08-2e86-4f49-be57-848ace5958b1-home\") pod \"simulator-w0-1529dc-kserve-8466d8d645-l54cf\" (UID: \"257edb08-2e86-4f49-be57-848ace5958b1\") " pod="e2e-models-e2e-worker-w0-1529dc/simulator-w0-1529dc-kserve-8466d8d645-l54cf" Oct 30 21:50:38.396256 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:38.396158 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpglj\" (UniqueName: \"kubernetes.io/projected/257edb08-2e86-4f49-be57-848ace5958b1-kube-api-access-wpglj\") pod \"simulator-w0-1529dc-kserve-8466d8d645-l54cf\" (UID: \"257edb08-2e86-4f49-be57-848ace5958b1\") " pod="e2e-models-e2e-worker-w0-1529dc/simulator-w0-1529dc-kserve-8466d8d645-l54cf" Oct 30 21:50:38.396256 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:38.396177 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/257edb08-2e86-4f49-be57-848ace5958b1-tmp-dir\") pod \"simulator-w0-1529dc-kserve-8466d8d645-l54cf\" (UID: \"257edb08-2e86-4f49-be57-848ace5958b1\") " pod="e2e-models-e2e-worker-w0-1529dc/simulator-w0-1529dc-kserve-8466d8d645-l54cf" Oct 30 21:50:38.396256 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:38.396193 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/257edb08-2e86-4f49-be57-848ace5958b1-tls-certs\") pod \"simulator-w0-1529dc-kserve-8466d8d645-l54cf\" (UID: \"257edb08-2e86-4f49-be57-848ace5958b1\") " pod="e2e-models-e2e-worker-w0-1529dc/simulator-w0-1529dc-kserve-8466d8d645-l54cf" Oct 30 21:50:38.396576 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:38.396556 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/257edb08-2e86-4f49-be57-848ace5958b1-home\") pod \"simulator-w0-1529dc-kserve-8466d8d645-l54cf\" (UID: \"257edb08-2e86-4f49-be57-848ace5958b1\") " pod="e2e-models-e2e-worker-w0-1529dc/simulator-w0-1529dc-kserve-8466d8d645-l54cf" Oct 30 21:50:38.396643 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:38.396612 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/257edb08-2e86-4f49-be57-848ace5958b1-model-cache\") pod \"simulator-w0-1529dc-kserve-8466d8d645-l54cf\" (UID: \"257edb08-2e86-4f49-be57-848ace5958b1\") " pod="e2e-models-e2e-worker-w0-1529dc/simulator-w0-1529dc-kserve-8466d8d645-l54cf" Oct 30 21:50:38.396682 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:38.396657 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/257edb08-2e86-4f49-be57-848ace5958b1-tmp-dir\") pod \"simulator-w0-1529dc-kserve-8466d8d645-l54cf\" (UID: \"257edb08-2e86-4f49-be57-848ace5958b1\") " pod="e2e-models-e2e-worker-w0-1529dc/simulator-w0-1529dc-kserve-8466d8d645-l54cf" Oct 30 21:50:38.398403 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:38.398369 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/257edb08-2e86-4f49-be57-848ace5958b1-dshm\") pod \"simulator-w0-1529dc-kserve-8466d8d645-l54cf\" (UID: \"257edb08-2e86-4f49-be57-848ace5958b1\") " pod="e2e-models-e2e-worker-w0-1529dc/simulator-w0-1529dc-kserve-8466d8d645-l54cf" Oct 30 21:50:38.398672 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:38.398655 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/257edb08-2e86-4f49-be57-848ace5958b1-tls-certs\") pod \"simulator-w0-1529dc-kserve-8466d8d645-l54cf\" (UID: \"257edb08-2e86-4f49-be57-848ace5958b1\") " pod="e2e-models-e2e-worker-w0-1529dc/simulator-w0-1529dc-kserve-8466d8d645-l54cf" Oct 30 21:50:38.414201 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:38.414175 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpglj\" (UniqueName: \"kubernetes.io/projected/257edb08-2e86-4f49-be57-848ace5958b1-kube-api-access-wpglj\") pod \"simulator-w0-1529dc-kserve-8466d8d645-l54cf\" (UID: \"257edb08-2e86-4f49-be57-848ace5958b1\") " pod="e2e-models-e2e-worker-w0-1529dc/simulator-w0-1529dc-kserve-8466d8d645-l54cf" Oct 30 21:50:38.531446 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:38.531345 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1529dc/simulator-w0-1529dc-kserve-8466d8d645-l54cf" Oct 30 21:50:38.818451 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:38.818419 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1529dc/simulator-w0-1529dc-kserve-8466d8d645-l54cf"] Oct 30 21:50:38.840485 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:38.840450 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-939a16/simulator-w4-939a16-kserve-74f845bf84-4c7b6"] Oct 30 21:50:38.845826 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:38.845804 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-939a16/simulator-w4-939a16-kserve-74f845bf84-4c7b6" Oct 30 21:50:38.853326 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:38.853299 6720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-939a16\"/\"openshift-service-ca.crt\"" Oct 30 21:50:38.856234 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:38.856214 6720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-939a16\"/\"kube-root-ca.crt\"" Oct 30 21:50:38.856346 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:38.856251 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-939a16/simulator-w4-939a16-kserve-74f845bf84-4c7b6"] Oct 30 21:50:38.856346 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:38.856311 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-939a16\"/\"simulator-w4-939a16-kserve-self-signed-certs\"" Oct 30 21:50:38.856464 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:38.856351 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-939a16\"/\"default-dockercfg-tsrf9\"" Oct 30 21:50:38.901722 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:38.901691 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bb6805f-f355-4ea8-80b7-542215c42022-dshm\") pod \"simulator-w4-939a16-kserve-74f845bf84-4c7b6\" (UID: \"9bb6805f-f355-4ea8-80b7-542215c42022\") " pod="e2e-models-e2e-worker-w4-939a16/simulator-w4-939a16-kserve-74f845bf84-4c7b6" Oct 30 21:50:38.901722 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:38.901725 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bb6805f-f355-4ea8-80b7-542215c42022-home\") pod \"simulator-w4-939a16-kserve-74f845bf84-4c7b6\" (UID: \"9bb6805f-f355-4ea8-80b7-542215c42022\") " pod="e2e-models-e2e-worker-w4-939a16/simulator-w4-939a16-kserve-74f845bf84-4c7b6" Oct 30 21:50:38.901981 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:38.901766 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bb6805f-f355-4ea8-80b7-542215c42022-tls-certs\") pod \"simulator-w4-939a16-kserve-74f845bf84-4c7b6\" (UID: \"9bb6805f-f355-4ea8-80b7-542215c42022\") " pod="e2e-models-e2e-worker-w4-939a16/simulator-w4-939a16-kserve-74f845bf84-4c7b6" Oct 30 21:50:38.901981 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:38.901911 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrxwm\" (UniqueName: \"kubernetes.io/projected/9bb6805f-f355-4ea8-80b7-542215c42022-kube-api-access-zrxwm\") pod \"simulator-w4-939a16-kserve-74f845bf84-4c7b6\" (UID: \"9bb6805f-f355-4ea8-80b7-542215c42022\") " pod="e2e-models-e2e-worker-w4-939a16/simulator-w4-939a16-kserve-74f845bf84-4c7b6" Oct 30 21:50:38.901981 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:38.901975 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bb6805f-f355-4ea8-80b7-542215c42022-tmp-dir\") pod \"simulator-w4-939a16-kserve-74f845bf84-4c7b6\" (UID: \"9bb6805f-f355-4ea8-80b7-542215c42022\") " pod="e2e-models-e2e-worker-w4-939a16/simulator-w4-939a16-kserve-74f845bf84-4c7b6" Oct 30 21:50:38.902130 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:38.902107 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bb6805f-f355-4ea8-80b7-542215c42022-model-cache\") pod \"simulator-w4-939a16-kserve-74f845bf84-4c7b6\" (UID: \"9bb6805f-f355-4ea8-80b7-542215c42022\") " pod="e2e-models-e2e-worker-w4-939a16/simulator-w4-939a16-kserve-74f845bf84-4c7b6" Oct 30 21:50:38.931781 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:38.931749 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:39.003673 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:39.003639 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bb6805f-f355-4ea8-80b7-542215c42022-tls-certs\") pod \"simulator-w4-939a16-kserve-74f845bf84-4c7b6\" (UID: \"9bb6805f-f355-4ea8-80b7-542215c42022\") " pod="e2e-models-e2e-worker-w4-939a16/simulator-w4-939a16-kserve-74f845bf84-4c7b6" Oct 30 21:50:39.003673 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:39.003682 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrxwm\" (UniqueName: \"kubernetes.io/projected/9bb6805f-f355-4ea8-80b7-542215c42022-kube-api-access-zrxwm\") pod \"simulator-w4-939a16-kserve-74f845bf84-4c7b6\" (UID: \"9bb6805f-f355-4ea8-80b7-542215c42022\") " pod="e2e-models-e2e-worker-w4-939a16/simulator-w4-939a16-kserve-74f845bf84-4c7b6" Oct 30 21:50:39.003938 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:39.003909 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bb6805f-f355-4ea8-80b7-542215c42022-tmp-dir\") pod \"simulator-w4-939a16-kserve-74f845bf84-4c7b6\" (UID: \"9bb6805f-f355-4ea8-80b7-542215c42022\") " pod="e2e-models-e2e-worker-w4-939a16/simulator-w4-939a16-kserve-74f845bf84-4c7b6" Oct 30 21:50:39.004610 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:39.004062 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bb6805f-f355-4ea8-80b7-542215c42022-model-cache\") pod \"simulator-w4-939a16-kserve-74f845bf84-4c7b6\" (UID: \"9bb6805f-f355-4ea8-80b7-542215c42022\") " pod="e2e-models-e2e-worker-w4-939a16/simulator-w4-939a16-kserve-74f845bf84-4c7b6" Oct 30 21:50:39.004610 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:39.004158 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bb6805f-f355-4ea8-80b7-542215c42022-dshm\") pod \"simulator-w4-939a16-kserve-74f845bf84-4c7b6\" (UID: \"9bb6805f-f355-4ea8-80b7-542215c42022\") " pod="e2e-models-e2e-worker-w4-939a16/simulator-w4-939a16-kserve-74f845bf84-4c7b6" Oct 30 21:50:39.004610 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:39.004185 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bb6805f-f355-4ea8-80b7-542215c42022-home\") pod \"simulator-w4-939a16-kserve-74f845bf84-4c7b6\" (UID: \"9bb6805f-f355-4ea8-80b7-542215c42022\") " pod="e2e-models-e2e-worker-w4-939a16/simulator-w4-939a16-kserve-74f845bf84-4c7b6" Oct 30 21:50:39.004610 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:39.004307 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bb6805f-f355-4ea8-80b7-542215c42022-tmp-dir\") pod \"simulator-w4-939a16-kserve-74f845bf84-4c7b6\" (UID: \"9bb6805f-f355-4ea8-80b7-542215c42022\") " pod="e2e-models-e2e-worker-w4-939a16/simulator-w4-939a16-kserve-74f845bf84-4c7b6" Oct 30 21:50:39.004610 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:39.004568 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bb6805f-f355-4ea8-80b7-542215c42022-model-cache\") pod \"simulator-w4-939a16-kserve-74f845bf84-4c7b6\" (UID: \"9bb6805f-f355-4ea8-80b7-542215c42022\") " pod="e2e-models-e2e-worker-w4-939a16/simulator-w4-939a16-kserve-74f845bf84-4c7b6" Oct 30 21:50:39.004926 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:39.004618 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bb6805f-f355-4ea8-80b7-542215c42022-home\") pod \"simulator-w4-939a16-kserve-74f845bf84-4c7b6\" (UID: \"9bb6805f-f355-4ea8-80b7-542215c42022\") " pod="e2e-models-e2e-worker-w4-939a16/simulator-w4-939a16-kserve-74f845bf84-4c7b6" Oct 30 21:50:39.006582 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:39.006556 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bb6805f-f355-4ea8-80b7-542215c42022-dshm\") pod \"simulator-w4-939a16-kserve-74f845bf84-4c7b6\" (UID: \"9bb6805f-f355-4ea8-80b7-542215c42022\") " pod="e2e-models-e2e-worker-w4-939a16/simulator-w4-939a16-kserve-74f845bf84-4c7b6" Oct 30 21:50:39.006828 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:39.006810 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bb6805f-f355-4ea8-80b7-542215c42022-tls-certs\") pod \"simulator-w4-939a16-kserve-74f845bf84-4c7b6\" (UID: \"9bb6805f-f355-4ea8-80b7-542215c42022\") " pod="e2e-models-e2e-worker-w4-939a16/simulator-w4-939a16-kserve-74f845bf84-4c7b6" Oct 30 21:50:39.017778 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:39.017740 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrxwm\" (UniqueName: \"kubernetes.io/projected/9bb6805f-f355-4ea8-80b7-542215c42022-kube-api-access-zrxwm\") pod \"simulator-w4-939a16-kserve-74f845bf84-4c7b6\" (UID: \"9bb6805f-f355-4ea8-80b7-542215c42022\") " pod="e2e-models-e2e-worker-w4-939a16/simulator-w4-939a16-kserve-74f845bf84-4c7b6" Oct 30 21:50:39.067307 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:39.067271 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:39.162459 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:39.162422 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-939a16/simulator-w4-939a16-kserve-74f845bf84-4c7b6" Oct 30 21:50:39.441129 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:39.441056 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-939a16/simulator-w4-939a16-kserve-74f845bf84-4c7b6"] Oct 30 21:50:39.444205 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:50:39.444179 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9bb6805f_f355_4ea8_80b7_542215c42022.slice/crio-6a6b15e4d09a0b7fac5ccdd62c8c069a531b25fdf5e70df69472d061d534c4b4 WatchSource:0}: Error finding container 6a6b15e4d09a0b7fac5ccdd62c8c069a531b25fdf5e70df69472d061d534c4b4: Status 404 returned error can't find the container with id 6a6b15e4d09a0b7fac5ccdd62c8c069a531b25fdf5e70df69472d061d534c4b4 Oct 30 21:50:39.771514 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:39.771423 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-939a16/simulator-w4-939a16-kserve-74f845bf84-4c7b6" event={"ID":"9bb6805f-f355-4ea8-80b7-542215c42022","Type":"ContainerStarted","Data":"6a6b15e4d09a0b7fac5ccdd62c8c069a531b25fdf5e70df69472d061d534c4b4"} Oct 30 21:50:39.773100 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:39.773073 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1529dc/simulator-w0-1529dc-kserve-8466d8d645-l54cf" event={"ID":"257edb08-2e86-4f49-be57-848ace5958b1","Type":"ContainerStarted","Data":"516f8e92dc61ceaa3d1867e38b6ea4ae17b9029b85c0543294711ab6740b88a6"} Oct 30 21:50:39.803414 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:39.803268 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:40.418184 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:50:40.418142 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 21:50:40.779574 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:40.779536 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1529dc/simulator-w0-1529dc-kserve-8466d8d645-l54cf" event={"ID":"257edb08-2e86-4f49-be57-848ace5958b1","Type":"ContainerStarted","Data":"741f2b751312455dd9ef9940876e36b70c91bd5c78d59a00ce290bd0014f4a02"} Oct 30 21:50:40.781430 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:40.781372 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-939a16/simulator-w4-939a16-kserve-74f845bf84-4c7b6" event={"ID":"9bb6805f-f355-4ea8-80b7-542215c42022","Type":"ContainerStarted","Data":"fb9912a8eb18f42b39c59f14b1b4c1ea131db15d2f7569520a8eb46d6cc51fde"} Oct 30 21:50:40.801931 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:40.801882 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-1529dc/simulator-w0-1529dc-kserve-8466d8d645-l54cf" podStartSLOduration=1.274578782 podStartE2EDuration="2.801865962s" totalImagesPullingTime="1.52728718s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:40.801286426 +0000 UTC m=+606.972251608" watchObservedRunningTime="2026-10-30 21:50:40.801865962 +0000 UTC m=+606.972831147" Oct 30 21:50:40.803546 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:40.803521 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:41.162553 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:41.162516 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-939a16/simulator-w4-939a16-kserve-74f845bf84-4c7b6" Oct 30 21:50:41.531679 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:41.531586 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1529dc/simulator-w0-1529dc-kserve-8466d8d645-l54cf" Oct 30 21:50:41.536560 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:41.536535 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:42.323353 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:42.323313 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:42.551970 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:42.551929 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:43.011084 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:43.011053 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:43.765899 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:43.765866 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:44.560971 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:44.560937 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:45.219232 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:45.219174 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:45.981932 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:45.981895 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:46.772887 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:46.772860 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:47.640045 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:47.640006 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:48.440594 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:48.440567 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:48.532413 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:48.532364 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-1529dc/simulator-w0-1529dc-kserve-8466d8d645-l54cf" Oct 30 21:50:48.545007 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:48.544984 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-1529dc/simulator-w0-1529dc-kserve-8466d8d645-l54cf" Oct 30 21:50:48.567166 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:48.567122 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-939a16/simulator-w4-939a16-kserve-74f845bf84-4c7b6" podStartSLOduration=9.402692974 podStartE2EDuration="10.567108286s" totalImagesPullingTime="1.164415312s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:40.825834584 +0000 UTC m=+606.996799766" watchObservedRunningTime="2026-10-30 21:50:48.567108286 +0000 UTC m=+614.738073701" Oct 30 21:50:48.842119 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:48.842036 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1529dc/simulator-w0-1529dc-kserve-8466d8d645-l54cf" Oct 30 21:50:48.917051 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:48.917005 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:49.141838 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:49.141798 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:49.163050 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:49.163014 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-939a16/simulator-w4-939a16-kserve-74f845bf84-4c7b6" Oct 30 21:50:49.178010 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:49.177985 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-939a16/simulator-w4-939a16-kserve-74f845bf84-4c7b6" Oct 30 21:50:49.849713 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:49.849676 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-939a16/simulator-w4-939a16-kserve-74f845bf84-4c7b6" Oct 30 21:50:49.919601 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:49.919549 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:50.625989 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:50.625951 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:51.350731 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:51.350696 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:52.233030 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:52.232998 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:52.935171 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:52.935114 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:53.628393 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:53.628352 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:54.342342 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:54.342313 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:54.999136 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:54.999097 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-ceb36a/premium-w3-ceb36a-kserve-6576bb476f-j5kdr"] Oct 30 21:50:55.003371 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:55.003344 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ceb36a/premium-w3-ceb36a-kserve-6576bb476f-j5kdr" Oct 30 21:50:55.015022 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:55.015002 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-ceb36a\"/\"default-dockercfg-86s4f\"" Oct 30 21:50:55.015334 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:55.015299 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-ceb36a\"/\"premium-w3-ceb36a-kserve-self-signed-certs\"" Oct 30 21:50:55.017628 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:55.017607 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-ceb36a/premium-w3-ceb36a-kserve-6576bb476f-j5kdr"] Oct 30 21:50:55.033209 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:55.033165 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:55.066975 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:55.066944 6720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-ceb36a\"/\"openshift-service-ca.crt\"" Oct 30 21:50:55.067179 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:55.067162 6720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-ceb36a\"/\"kube-root-ca.crt\"" Oct 30 21:50:55.087704 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:55.087671 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d065e788-c972-4683-84ed-86a70ed4791d-home\") pod \"premium-w3-ceb36a-kserve-6576bb476f-j5kdr\" (UID: \"d065e788-c972-4683-84ed-86a70ed4791d\") " pod="e2e-models-e2e-worker-w3-ceb36a/premium-w3-ceb36a-kserve-6576bb476f-j5kdr" Oct 30 21:50:55.087841 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:55.087708 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-527xz\" (UniqueName: \"kubernetes.io/projected/d065e788-c972-4683-84ed-86a70ed4791d-kube-api-access-527xz\") pod \"premium-w3-ceb36a-kserve-6576bb476f-j5kdr\" (UID: \"d065e788-c972-4683-84ed-86a70ed4791d\") " pod="e2e-models-e2e-worker-w3-ceb36a/premium-w3-ceb36a-kserve-6576bb476f-j5kdr" Oct 30 21:50:55.087841 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:55.087749 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d065e788-c972-4683-84ed-86a70ed4791d-dshm\") pod \"premium-w3-ceb36a-kserve-6576bb476f-j5kdr\" (UID: \"d065e788-c972-4683-84ed-86a70ed4791d\") " pod="e2e-models-e2e-worker-w3-ceb36a/premium-w3-ceb36a-kserve-6576bb476f-j5kdr" Oct 30 21:50:55.087918 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:55.087876 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d065e788-c972-4683-84ed-86a70ed4791d-model-cache\") pod \"premium-w3-ceb36a-kserve-6576bb476f-j5kdr\" (UID: \"d065e788-c972-4683-84ed-86a70ed4791d\") " pod="e2e-models-e2e-worker-w3-ceb36a/premium-w3-ceb36a-kserve-6576bb476f-j5kdr" Oct 30 21:50:55.087918 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:55.087905 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d065e788-c972-4683-84ed-86a70ed4791d-tls-certs\") pod \"premium-w3-ceb36a-kserve-6576bb476f-j5kdr\" (UID: \"d065e788-c972-4683-84ed-86a70ed4791d\") " pod="e2e-models-e2e-worker-w3-ceb36a/premium-w3-ceb36a-kserve-6576bb476f-j5kdr" Oct 30 21:50:55.088053 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:55.088030 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d065e788-c972-4683-84ed-86a70ed4791d-tmp-dir\") pod \"premium-w3-ceb36a-kserve-6576bb476f-j5kdr\" (UID: \"d065e788-c972-4683-84ed-86a70ed4791d\") " pod="e2e-models-e2e-worker-w3-ceb36a/premium-w3-ceb36a-kserve-6576bb476f-j5kdr" Oct 30 21:50:55.188663 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:55.188619 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-527xz\" (UniqueName: \"kubernetes.io/projected/d065e788-c972-4683-84ed-86a70ed4791d-kube-api-access-527xz\") pod \"premium-w3-ceb36a-kserve-6576bb476f-j5kdr\" (UID: \"d065e788-c972-4683-84ed-86a70ed4791d\") " pod="e2e-models-e2e-worker-w3-ceb36a/premium-w3-ceb36a-kserve-6576bb476f-j5kdr" Oct 30 21:50:55.188870 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:55.188680 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d065e788-c972-4683-84ed-86a70ed4791d-dshm\") pod \"premium-w3-ceb36a-kserve-6576bb476f-j5kdr\" (UID: \"d065e788-c972-4683-84ed-86a70ed4791d\") " pod="e2e-models-e2e-worker-w3-ceb36a/premium-w3-ceb36a-kserve-6576bb476f-j5kdr" Oct 30 21:50:55.188870 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:55.188719 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d065e788-c972-4683-84ed-86a70ed4791d-model-cache\") pod \"premium-w3-ceb36a-kserve-6576bb476f-j5kdr\" (UID: \"d065e788-c972-4683-84ed-86a70ed4791d\") " pod="e2e-models-e2e-worker-w3-ceb36a/premium-w3-ceb36a-kserve-6576bb476f-j5kdr" Oct 30 21:50:55.188870 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:55.188739 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d065e788-c972-4683-84ed-86a70ed4791d-tls-certs\") pod \"premium-w3-ceb36a-kserve-6576bb476f-j5kdr\" (UID: \"d065e788-c972-4683-84ed-86a70ed4791d\") " pod="e2e-models-e2e-worker-w3-ceb36a/premium-w3-ceb36a-kserve-6576bb476f-j5kdr" Oct 30 21:50:55.188870 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:55.188801 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d065e788-c972-4683-84ed-86a70ed4791d-tmp-dir\") pod \"premium-w3-ceb36a-kserve-6576bb476f-j5kdr\" (UID: \"d065e788-c972-4683-84ed-86a70ed4791d\") " pod="e2e-models-e2e-worker-w3-ceb36a/premium-w3-ceb36a-kserve-6576bb476f-j5kdr" Oct 30 21:50:55.188870 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:55.188836 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d065e788-c972-4683-84ed-86a70ed4791d-home\") pod \"premium-w3-ceb36a-kserve-6576bb476f-j5kdr\" (UID: \"d065e788-c972-4683-84ed-86a70ed4791d\") " pod="e2e-models-e2e-worker-w3-ceb36a/premium-w3-ceb36a-kserve-6576bb476f-j5kdr" Oct 30 21:50:55.189232 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:55.189208 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d065e788-c972-4683-84ed-86a70ed4791d-model-cache\") pod \"premium-w3-ceb36a-kserve-6576bb476f-j5kdr\" (UID: \"d065e788-c972-4683-84ed-86a70ed4791d\") " pod="e2e-models-e2e-worker-w3-ceb36a/premium-w3-ceb36a-kserve-6576bb476f-j5kdr" Oct 30 21:50:55.189301 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:55.189226 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d065e788-c972-4683-84ed-86a70ed4791d-home\") pod \"premium-w3-ceb36a-kserve-6576bb476f-j5kdr\" (UID: \"d065e788-c972-4683-84ed-86a70ed4791d\") " pod="e2e-models-e2e-worker-w3-ceb36a/premium-w3-ceb36a-kserve-6576bb476f-j5kdr" Oct 30 21:50:55.189301 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:55.189270 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d065e788-c972-4683-84ed-86a70ed4791d-tmp-dir\") pod \"premium-w3-ceb36a-kserve-6576bb476f-j5kdr\" (UID: \"d065e788-c972-4683-84ed-86a70ed4791d\") " pod="e2e-models-e2e-worker-w3-ceb36a/premium-w3-ceb36a-kserve-6576bb476f-j5kdr" Oct 30 21:50:55.191056 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:55.191036 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d065e788-c972-4683-84ed-86a70ed4791d-dshm\") pod \"premium-w3-ceb36a-kserve-6576bb476f-j5kdr\" (UID: \"d065e788-c972-4683-84ed-86a70ed4791d\") " pod="e2e-models-e2e-worker-w3-ceb36a/premium-w3-ceb36a-kserve-6576bb476f-j5kdr" Oct 30 21:50:55.191303 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:55.191280 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d065e788-c972-4683-84ed-86a70ed4791d-tls-certs\") pod \"premium-w3-ceb36a-kserve-6576bb476f-j5kdr\" (UID: \"d065e788-c972-4683-84ed-86a70ed4791d\") " pod="e2e-models-e2e-worker-w3-ceb36a/premium-w3-ceb36a-kserve-6576bb476f-j5kdr" Oct 30 21:50:55.201560 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:55.201525 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:55.202888 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:55.202865 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-527xz\" (UniqueName: \"kubernetes.io/projected/d065e788-c972-4683-84ed-86a70ed4791d-kube-api-access-527xz\") pod \"premium-w3-ceb36a-kserve-6576bb476f-j5kdr\" (UID: \"d065e788-c972-4683-84ed-86a70ed4791d\") " pod="e2e-models-e2e-worker-w3-ceb36a/premium-w3-ceb36a-kserve-6576bb476f-j5kdr" Oct 30 21:50:55.318916 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:55.318832 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ceb36a/premium-w3-ceb36a-kserve-6576bb476f-j5kdr" Oct 30 21:50:55.648024 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:55.648002 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-ceb36a/premium-w3-ceb36a-kserve-6576bb476f-j5kdr"] Oct 30 21:50:55.650173 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:50:55.650136 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd065e788_c972_4683_84ed_86a70ed4791d.slice/crio-cf5bc4a1325d7787b775673abbd83daad39839106a378d0c43fbe6d215764cb6 WatchSource:0}: Error finding container cf5bc4a1325d7787b775673abbd83daad39839106a378d0c43fbe6d215764cb6: Status 404 returned error can't find the container with id cf5bc4a1325d7787b775673abbd83daad39839106a378d0c43fbe6d215764cb6 Oct 30 21:50:55.830977 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:55.830929 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:55.879780 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:55.879736 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ceb36a/premium-w3-ceb36a-kserve-6576bb476f-j5kdr" event={"ID":"d065e788-c972-4683-84ed-86a70ed4791d","Type":"ContainerStarted","Data":"01aa584df2fa7f4f1f95b34835fe0ec63e61de670771b7f974407d3033d75cb0"} Oct 30 21:50:55.879957 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:55.879787 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ceb36a/premium-w3-ceb36a-kserve-6576bb476f-j5kdr" event={"ID":"d065e788-c972-4683-84ed-86a70ed4791d","Type":"ContainerStarted","Data":"cf5bc4a1325d7787b775673abbd83daad39839106a378d0c43fbe6d215764cb6"} Oct 30 21:50:55.904624 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:55.904537 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-ceb36a/premium-w3-ceb36a-kserve-6576bb476f-j5kdr" podStartSLOduration=1.9045236380000001 podStartE2EDuration="1.904523638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:55.901875165 +0000 UTC m=+622.072840347" watchObservedRunningTime="2026-10-30 21:50:55.904523638 +0000 UTC m=+622.075488821" Oct 30 21:50:56.319183 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:56.319103 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ceb36a/premium-w3-ceb36a-kserve-6576bb476f-j5kdr" Oct 30 21:50:56.636323 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:56.636270 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:57.313106 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:57.313076 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:58.144615 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:58.144580 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:58.836146 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:58.836108 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:58.986272 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:58.986241 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:50:59.634571 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:50:59.634537 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:00.416608 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:51:00.416575 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:01.110296 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:51:01.110265 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:01.827237 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:51:01.827201 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:02.505764 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:51:02.505732 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:03.224494 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:51:03.224443 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:03.914247 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:51:03.914212 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:04.628631 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:51:04.628602 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:05.078664 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:51:05.078581 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:05.319979 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:51:05.319946 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-ceb36a/premium-w3-ceb36a-kserve-6576bb476f-j5kdr" Oct 30 21:51:05.335072 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:51:05.334999 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-ceb36a/premium-w3-ceb36a-kserve-6576bb476f-j5kdr" Oct 30 21:51:05.622733 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:51:05.622700 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:05.946085 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:51:05.946010 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ceb36a/premium-w3-ceb36a-kserve-6576bb476f-j5kdr" Oct 30 21:51:06.525365 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:51:06.525324 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:07.209968 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:51:07.209935 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:07.832034 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:51:07.831996 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:08.546750 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:51:08.546711 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:08.687118 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:51:08.687081 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:09.321312 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:51:09.321281 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:10.217714 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:51:10.217679 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:10.403798 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:51:10.403759 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 21:51:11.135290 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:51:11.135249 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:11.867541 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:51:11.867507 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:12.527067 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:51:12.527039 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:13.223892 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:51:13.223837 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:13.915083 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:51:13.915044 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:14.634882 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:51:14.634802 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:15.452938 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:51:15.452903 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:16.116818 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:51:16.116787 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:16.839058 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:51:16.839018 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:17.745122 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:51:17.745052 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:18.456783 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:51:18.456744 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:19.342948 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:51:19.342908 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:20.148104 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:51:20.148063 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:21.205157 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:51:21.205121 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:21.940741 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:51:21.940689 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:22.851625 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:51:22.851589 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:23.526249 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:51:23.526197 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:24.229030 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:51:24.229002 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:24.913961 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:51:24.913930 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:25.579074 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:51:25.579042 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:26.112969 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:51:26.112890 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:26.442551 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:51:26.442455 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:27.119865 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:51:27.119828 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:27.732939 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:51:27.732907 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:28.425297 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:51:28.425266 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:29.096924 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:51:29.096894 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:29.880073 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:51:29.879978 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:30.747408 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:51:30.747148 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:31.718859 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:51:31.718820 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:32.578608 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:51:32.578554 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:33.218902 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:51:33.218870 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:33.902435 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:51:33.902368 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:34.645109 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:51:34.645063 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:35.236123 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:51:35.236088 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:35.914220 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:51:35.914176 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:36.627902 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:51:36.627861 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:37.305505 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:51:37.305466 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:38.017372 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:51:38.017295 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:38.819864 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:51:38.819828 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:39.719342 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:51:39.719312 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:40.311055 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:51:40.311019 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:40.401903 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:51:40.401869 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 21:51:40.532428 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:51:40.532353 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:41.332209 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:51:41.332178 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:42.100296 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:51:42.100258 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:42.846813 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:51:42.846777 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:43.554924 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:51:43.554892 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:44.234717 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:51:44.234684 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:44.921778 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:51:44.921743 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:45.839318 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:51:45.839289 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:46.531952 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:51:46.531914 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:47.347886 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:51:47.347842 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:48.021808 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:51:48.021765 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:48.721057 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:51:48.721023 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:49.607228 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:51:49.607141 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:50.423905 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:51:50.423873 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:51.142687 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:51:51.142648 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:51.817319 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:51:51.817278 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:52.707267 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:51:52.707158 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:53.349844 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:51:53.349814 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:54.035643 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:51:54.035586 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:54.321672 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:51:54.321602 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:54.616030 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:51:54.615999 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:55.028954 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:51:55.028880 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:55.825128 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:51:55.825092 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:56.699806 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:51:56.699772 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:57.830071 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:51:57.829992 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:58.523351 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:51:58.523319 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:51:59.369270 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:51:59.369230 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:00.421233 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:00.421202 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:01.416282 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:01.416248 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:02.431033 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:02.430988 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:03.125474 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:03.125435 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:03.925681 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:03.925653 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:04.639104 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:04.639075 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:05.347030 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:05.346994 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:06.112887 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:06.112848 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:06.829087 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:06.829053 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:07.581816 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:07.581782 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:08.210737 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:08.210684 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:09.022200 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:09.022167 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:09.739761 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:09.739728 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:10.404746 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:52:10.404708 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 21:52:10.462255 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:10.462228 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:11.044237 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:11.044206 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-ceb36a/e2e-conflict-a-dfac3fd4-kserve-677cc88846-vf5kf"] Oct 30 21:52:11.054301 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:11.054270 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ceb36a/e2e-conflict-a-dfac3fd4-kserve-677cc88846-vf5kf" Oct 30 21:52:11.064249 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:11.064204 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-ceb36a\"/\"e2e-conflict-a-dfac3fd4-kserve-self-signed-certs\"" Oct 30 21:52:11.066220 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:11.066191 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-ceb36a/e2e-conflict-a-dfac3fd4-kserve-677cc88846-vf5kf"] Oct 30 21:52:11.112623 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:11.112587 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab5d437e-6b94-4c23-b45e-a93f8a840ffe-dshm\") pod \"e2e-conflict-a-dfac3fd4-kserve-677cc88846-vf5kf\" (UID: \"ab5d437e-6b94-4c23-b45e-a93f8a840ffe\") " pod="e2e-models-e2e-worker-w3-ceb36a/e2e-conflict-a-dfac3fd4-kserve-677cc88846-vf5kf" Oct 30 21:52:11.112781 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:11.112651 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab5d437e-6b94-4c23-b45e-a93f8a840ffe-model-cache\") pod \"e2e-conflict-a-dfac3fd4-kserve-677cc88846-vf5kf\" (UID: \"ab5d437e-6b94-4c23-b45e-a93f8a840ffe\") " pod="e2e-models-e2e-worker-w3-ceb36a/e2e-conflict-a-dfac3fd4-kserve-677cc88846-vf5kf" Oct 30 21:52:11.112825 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:11.112810 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab5d437e-6b94-4c23-b45e-a93f8a840ffe-tmp-dir\") pod \"e2e-conflict-a-dfac3fd4-kserve-677cc88846-vf5kf\" (UID: \"ab5d437e-6b94-4c23-b45e-a93f8a840ffe\") " pod="e2e-models-e2e-worker-w3-ceb36a/e2e-conflict-a-dfac3fd4-kserve-677cc88846-vf5kf" Oct 30 21:52:11.112878 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:11.112863 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab5d437e-6b94-4c23-b45e-a93f8a840ffe-tls-certs\") pod \"e2e-conflict-a-dfac3fd4-kserve-677cc88846-vf5kf\" (UID: \"ab5d437e-6b94-4c23-b45e-a93f8a840ffe\") " pod="e2e-models-e2e-worker-w3-ceb36a/e2e-conflict-a-dfac3fd4-kserve-677cc88846-vf5kf" Oct 30 21:52:11.113041 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:11.113014 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab5d437e-6b94-4c23-b45e-a93f8a840ffe-home\") pod \"e2e-conflict-a-dfac3fd4-kserve-677cc88846-vf5kf\" (UID: \"ab5d437e-6b94-4c23-b45e-a93f8a840ffe\") " pod="e2e-models-e2e-worker-w3-ceb36a/e2e-conflict-a-dfac3fd4-kserve-677cc88846-vf5kf" Oct 30 21:52:11.113096 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:11.113079 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6rxq\" (UniqueName: \"kubernetes.io/projected/ab5d437e-6b94-4c23-b45e-a93f8a840ffe-kube-api-access-t6rxq\") pod \"e2e-conflict-a-dfac3fd4-kserve-677cc88846-vf5kf\" (UID: \"ab5d437e-6b94-4c23-b45e-a93f8a840ffe\") " pod="e2e-models-e2e-worker-w3-ceb36a/e2e-conflict-a-dfac3fd4-kserve-677cc88846-vf5kf" Oct 30 21:52:11.134620 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:11.134589 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:11.214607 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:11.214565 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab5d437e-6b94-4c23-b45e-a93f8a840ffe-tls-certs\") pod \"e2e-conflict-a-dfac3fd4-kserve-677cc88846-vf5kf\" (UID: \"ab5d437e-6b94-4c23-b45e-a93f8a840ffe\") " pod="e2e-models-e2e-worker-w3-ceb36a/e2e-conflict-a-dfac3fd4-kserve-677cc88846-vf5kf" Oct 30 21:52:11.214837 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:11.214665 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab5d437e-6b94-4c23-b45e-a93f8a840ffe-home\") pod \"e2e-conflict-a-dfac3fd4-kserve-677cc88846-vf5kf\" (UID: \"ab5d437e-6b94-4c23-b45e-a93f8a840ffe\") " pod="e2e-models-e2e-worker-w3-ceb36a/e2e-conflict-a-dfac3fd4-kserve-677cc88846-vf5kf" Oct 30 21:52:11.214837 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:11.214715 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6rxq\" (UniqueName: \"kubernetes.io/projected/ab5d437e-6b94-4c23-b45e-a93f8a840ffe-kube-api-access-t6rxq\") pod \"e2e-conflict-a-dfac3fd4-kserve-677cc88846-vf5kf\" (UID: \"ab5d437e-6b94-4c23-b45e-a93f8a840ffe\") " pod="e2e-models-e2e-worker-w3-ceb36a/e2e-conflict-a-dfac3fd4-kserve-677cc88846-vf5kf" Oct 30 21:52:11.214837 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:11.214822 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab5d437e-6b94-4c23-b45e-a93f8a840ffe-dshm\") pod \"e2e-conflict-a-dfac3fd4-kserve-677cc88846-vf5kf\" (UID: \"ab5d437e-6b94-4c23-b45e-a93f8a840ffe\") " pod="e2e-models-e2e-worker-w3-ceb36a/e2e-conflict-a-dfac3fd4-kserve-677cc88846-vf5kf" Oct 30 21:52:11.215015 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:11.214862 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab5d437e-6b94-4c23-b45e-a93f8a840ffe-model-cache\") pod \"e2e-conflict-a-dfac3fd4-kserve-677cc88846-vf5kf\" (UID: \"ab5d437e-6b94-4c23-b45e-a93f8a840ffe\") " pod="e2e-models-e2e-worker-w3-ceb36a/e2e-conflict-a-dfac3fd4-kserve-677cc88846-vf5kf" Oct 30 21:52:11.215015 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:11.214943 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab5d437e-6b94-4c23-b45e-a93f8a840ffe-tmp-dir\") pod \"e2e-conflict-a-dfac3fd4-kserve-677cc88846-vf5kf\" (UID: \"ab5d437e-6b94-4c23-b45e-a93f8a840ffe\") " pod="e2e-models-e2e-worker-w3-ceb36a/e2e-conflict-a-dfac3fd4-kserve-677cc88846-vf5kf" Oct 30 21:52:11.215247 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:11.215190 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab5d437e-6b94-4c23-b45e-a93f8a840ffe-home\") pod \"e2e-conflict-a-dfac3fd4-kserve-677cc88846-vf5kf\" (UID: \"ab5d437e-6b94-4c23-b45e-a93f8a840ffe\") " pod="e2e-models-e2e-worker-w3-ceb36a/e2e-conflict-a-dfac3fd4-kserve-677cc88846-vf5kf" Oct 30 21:52:11.215349 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:11.215330 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab5d437e-6b94-4c23-b45e-a93f8a840ffe-tmp-dir\") pod \"e2e-conflict-a-dfac3fd4-kserve-677cc88846-vf5kf\" (UID: \"ab5d437e-6b94-4c23-b45e-a93f8a840ffe\") " pod="e2e-models-e2e-worker-w3-ceb36a/e2e-conflict-a-dfac3fd4-kserve-677cc88846-vf5kf" Oct 30 21:52:11.215603 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:11.215580 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab5d437e-6b94-4c23-b45e-a93f8a840ffe-model-cache\") pod \"e2e-conflict-a-dfac3fd4-kserve-677cc88846-vf5kf\" (UID: \"ab5d437e-6b94-4c23-b45e-a93f8a840ffe\") " pod="e2e-models-e2e-worker-w3-ceb36a/e2e-conflict-a-dfac3fd4-kserve-677cc88846-vf5kf" Oct 30 21:52:11.217514 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:11.217492 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab5d437e-6b94-4c23-b45e-a93f8a840ffe-dshm\") pod \"e2e-conflict-a-dfac3fd4-kserve-677cc88846-vf5kf\" (UID: \"ab5d437e-6b94-4c23-b45e-a93f8a840ffe\") " pod="e2e-models-e2e-worker-w3-ceb36a/e2e-conflict-a-dfac3fd4-kserve-677cc88846-vf5kf" Oct 30 21:52:11.217636 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:11.217617 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab5d437e-6b94-4c23-b45e-a93f8a840ffe-tls-certs\") pod \"e2e-conflict-a-dfac3fd4-kserve-677cc88846-vf5kf\" (UID: \"ab5d437e-6b94-4c23-b45e-a93f8a840ffe\") " pod="e2e-models-e2e-worker-w3-ceb36a/e2e-conflict-a-dfac3fd4-kserve-677cc88846-vf5kf" Oct 30 21:52:11.227786 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:11.227766 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6rxq\" (UniqueName: \"kubernetes.io/projected/ab5d437e-6b94-4c23-b45e-a93f8a840ffe-kube-api-access-t6rxq\") pod \"e2e-conflict-a-dfac3fd4-kserve-677cc88846-vf5kf\" (UID: \"ab5d437e-6b94-4c23-b45e-a93f8a840ffe\") " pod="e2e-models-e2e-worker-w3-ceb36a/e2e-conflict-a-dfac3fd4-kserve-677cc88846-vf5kf" Oct 30 21:52:11.297967 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:11.297784 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:11.369316 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:11.369285 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ceb36a/e2e-conflict-a-dfac3fd4-kserve-677cc88846-vf5kf" Oct 30 21:52:11.653802 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:11.653777 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-ceb36a/e2e-conflict-a-dfac3fd4-kserve-677cc88846-vf5kf"] Oct 30 21:52:11.656303 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:52:11.656276 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podab5d437e_6b94_4c23_b45e_a93f8a840ffe.slice/crio-de06cf6404b78614377513ee259c55797246c1ba0086009683d9798462c00830 WatchSource:0}: Error finding container de06cf6404b78614377513ee259c55797246c1ba0086009683d9798462c00830: Status 404 returned error can't find the container with id de06cf6404b78614377513ee259c55797246c1ba0086009683d9798462c00830 Oct 30 21:52:11.939648 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:11.939570 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:12.317257 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:12.317171 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ceb36a/e2e-conflict-a-dfac3fd4-kserve-677cc88846-vf5kf" event={"ID":"ab5d437e-6b94-4c23-b45e-a93f8a840ffe","Type":"ContainerStarted","Data":"ef17724ea2f40cb6a5d394ab0b3c0195a34835bc73e2242fb6573518fe9bb00a"} Oct 30 21:52:12.317257 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:12.317213 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ceb36a/e2e-conflict-a-dfac3fd4-kserve-677cc88846-vf5kf" event={"ID":"ab5d437e-6b94-4c23-b45e-a93f8a840ffe","Type":"ContainerStarted","Data":"de06cf6404b78614377513ee259c55797246c1ba0086009683d9798462c00830"} Oct 30 21:52:12.338717 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:12.338668 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-ceb36a/e2e-conflict-a-dfac3fd4-kserve-677cc88846-vf5kf" podStartSLOduration=1.338655474 podStartE2EDuration="1.338655474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:12.336615708 +0000 UTC m=+698.507580901" watchObservedRunningTime="2026-10-30 21:52:12.338655474 +0000 UTC m=+698.509620656" Oct 30 21:52:12.370182 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:12.370149 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ceb36a/e2e-conflict-a-dfac3fd4-kserve-677cc88846-vf5kf" Oct 30 21:52:12.622285 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:12.622249 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:13.312088 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:13.312052 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:14.018941 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:14.018908 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:14.721456 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:14.721427 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:15.434080 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:15.434051 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:16.319777 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:16.319742 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:17.026534 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:17.026497 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:17.826148 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:17.826103 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:18.652064 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:18.652012 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:19.596979 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:19.596893 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:20.657694 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:20.657650 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:21.370452 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:21.370423 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-ceb36a/e2e-conflict-a-dfac3fd4-kserve-677cc88846-vf5kf" Oct 30 21:52:21.384395 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:21.384351 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-ceb36a/e2e-conflict-a-dfac3fd4-kserve-677cc88846-vf5kf" Oct 30 21:52:21.403659 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:21.403631 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ceb36a/e2e-conflict-a-dfac3fd4-kserve-677cc88846-vf5kf" Oct 30 21:52:21.423281 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:21.423252 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:22.277000 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:22.276968 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:22.564291 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:22.564209 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl"] Oct 30 21:52:22.568353 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:22.568332 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl" Oct 30 21:52:22.578318 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:22.578295 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-2b2e72-openshift-default-dockercfg-98gbf\"" Oct 30 21:52:22.590244 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:22.590221 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl"] Oct 30 21:52:22.665321 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:22.665287 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63d87167-397f-476b-837f-e40c8f14a994-istio-envoy\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl\" (UID: \"63d87167-397f-476b-837f-e40c8f14a994\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl" Oct 30 21:52:22.665321 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:22.665323 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63d87167-397f-476b-837f-e40c8f14a994-workload-certs\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl\" (UID: \"63d87167-397f-476b-837f-e40c8f14a994\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl" Oct 30 21:52:22.665598 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:22.665345 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63d87167-397f-476b-837f-e40c8f14a994-istiod-ca-cert\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl\" (UID: \"63d87167-397f-476b-837f-e40c8f14a994\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl" Oct 30 21:52:22.665598 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:22.665496 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63d87167-397f-476b-837f-e40c8f14a994-workload-socket\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl\" (UID: \"63d87167-397f-476b-837f-e40c8f14a994\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl" Oct 30 21:52:22.665598 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:22.665540 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63d87167-397f-476b-837f-e40c8f14a994-istio-data\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl\" (UID: \"63d87167-397f-476b-837f-e40c8f14a994\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl" Oct 30 21:52:22.665752 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:22.665639 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcfxt\" (UniqueName: \"kubernetes.io/projected/63d87167-397f-476b-837f-e40c8f14a994-kube-api-access-tcfxt\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl\" (UID: \"63d87167-397f-476b-837f-e40c8f14a994\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl" Oct 30 21:52:22.665752 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:22.665688 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63d87167-397f-476b-837f-e40c8f14a994-credential-socket\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl\" (UID: \"63d87167-397f-476b-837f-e40c8f14a994\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl" Oct 30 21:52:22.665752 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:22.665704 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63d87167-397f-476b-837f-e40c8f14a994-istio-token\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl\" (UID: \"63d87167-397f-476b-837f-e40c8f14a994\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl" Oct 30 21:52:22.665897 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:22.665838 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63d87167-397f-476b-837f-e40c8f14a994-istio-podinfo\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl\" (UID: \"63d87167-397f-476b-837f-e40c8f14a994\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl" Oct 30 21:52:22.766537 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:22.766503 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63d87167-397f-476b-837f-e40c8f14a994-workload-socket\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl\" (UID: \"63d87167-397f-476b-837f-e40c8f14a994\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl" Oct 30 21:52:22.766537 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:22.766544 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63d87167-397f-476b-837f-e40c8f14a994-istio-data\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl\" (UID: \"63d87167-397f-476b-837f-e40c8f14a994\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl" Oct 30 21:52:22.766760 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:22.766566 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcfxt\" (UniqueName: \"kubernetes.io/projected/63d87167-397f-476b-837f-e40c8f14a994-kube-api-access-tcfxt\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl\" (UID: \"63d87167-397f-476b-837f-e40c8f14a994\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl" Oct 30 21:52:22.766760 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:22.766596 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63d87167-397f-476b-837f-e40c8f14a994-credential-socket\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl\" (UID: \"63d87167-397f-476b-837f-e40c8f14a994\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl" Oct 30 21:52:22.766760 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:22.766617 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63d87167-397f-476b-837f-e40c8f14a994-istio-token\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl\" (UID: \"63d87167-397f-476b-837f-e40c8f14a994\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl" Oct 30 21:52:22.766760 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:22.766696 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63d87167-397f-476b-837f-e40c8f14a994-istio-podinfo\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl\" (UID: \"63d87167-397f-476b-837f-e40c8f14a994\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl" Oct 30 21:52:22.766760 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:22.766748 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63d87167-397f-476b-837f-e40c8f14a994-istio-envoy\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl\" (UID: \"63d87167-397f-476b-837f-e40c8f14a994\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl" Oct 30 21:52:22.767077 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:22.766775 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63d87167-397f-476b-837f-e40c8f14a994-workload-certs\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl\" (UID: \"63d87167-397f-476b-837f-e40c8f14a994\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl" Oct 30 21:52:22.767077 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:22.766801 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63d87167-397f-476b-837f-e40c8f14a994-istiod-ca-cert\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl\" (UID: \"63d87167-397f-476b-837f-e40c8f14a994\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl" Oct 30 21:52:22.767077 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:22.766960 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63d87167-397f-476b-837f-e40c8f14a994-istio-data\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl\" (UID: \"63d87167-397f-476b-837f-e40c8f14a994\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl" Oct 30 21:52:22.767077 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:22.766992 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63d87167-397f-476b-837f-e40c8f14a994-workload-socket\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl\" (UID: \"63d87167-397f-476b-837f-e40c8f14a994\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl" Oct 30 21:52:22.767274 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:22.767108 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63d87167-397f-476b-837f-e40c8f14a994-credential-socket\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl\" (UID: \"63d87167-397f-476b-837f-e40c8f14a994\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl" Oct 30 21:52:22.767274 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:22.767214 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63d87167-397f-476b-837f-e40c8f14a994-workload-certs\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl\" (UID: \"63d87167-397f-476b-837f-e40c8f14a994\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl" Oct 30 21:52:22.767506 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:22.767488 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63d87167-397f-476b-837f-e40c8f14a994-istiod-ca-cert\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl\" (UID: \"63d87167-397f-476b-837f-e40c8f14a994\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl" Oct 30 21:52:22.769321 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:22.769297 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63d87167-397f-476b-837f-e40c8f14a994-istio-envoy\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl\" (UID: \"63d87167-397f-476b-837f-e40c8f14a994\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl" Oct 30 21:52:22.769443 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:22.769335 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63d87167-397f-476b-837f-e40c8f14a994-istio-podinfo\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl\" (UID: \"63d87167-397f-476b-837f-e40c8f14a994\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl" Oct 30 21:52:22.779572 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:22.779545 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63d87167-397f-476b-837f-e40c8f14a994-istio-token\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl\" (UID: \"63d87167-397f-476b-837f-e40c8f14a994\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl" Oct 30 21:52:22.780200 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:22.780183 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcfxt\" (UniqueName: \"kubernetes.io/projected/63d87167-397f-476b-837f-e40c8f14a994-kube-api-access-tcfxt\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl\" (UID: \"63d87167-397f-476b-837f-e40c8f14a994\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl" Oct 30 21:52:22.883092 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:22.883060 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl" Oct 30 21:52:23.175345 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:23.175270 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl"] Oct 30 21:52:23.179481 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:52:23.179451 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63d87167_397f_476b_837f_e40c8f14a994.slice/crio-d9ca1487294c6ece545148dfb5e75c599d7425da7d45e443fa0a0f742462d656 WatchSource:0}: Error finding container d9ca1487294c6ece545148dfb5e75c599d7425da7d45e443fa0a0f742462d656: Status 404 returned error can't find the container with id d9ca1487294c6ece545148dfb5e75c599d7425da7d45e443fa0a0f742462d656 Oct 30 21:52:23.181833 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:23.181803 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:52:23.181913 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:23.181868 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:52:23.181959 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:23.181916 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:52:23.246431 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:23.246371 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:23.406620 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:23.405665 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl" event={"ID":"63d87167-397f-476b-837f-e40c8f14a994","Type":"ContainerStarted","Data":"f17f342c14597f7e83fac1551383c507e2739e0a757e7c889d2cd49a1f890edd"} Oct 30 21:52:23.406620 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:23.405730 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl" event={"ID":"63d87167-397f-476b-837f-e40c8f14a994","Type":"ContainerStarted","Data":"d9ca1487294c6ece545148dfb5e75c599d7425da7d45e443fa0a0f742462d656"} Oct 30 21:52:23.440590 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:23.440469 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl" podStartSLOduration=1.440451555 podStartE2EDuration="1.440451555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:23.43657931 +0000 UTC m=+709.607544493" watchObservedRunningTime="2026-10-30 21:52:23.440451555 +0000 UTC m=+709.611416738" Oct 30 21:52:23.466443 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:23.466412 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:23.883644 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:23.883597 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl" Oct 30 21:52:24.127498 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:24.127456 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:24.885441 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:24.885404 6720 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 30 21:52:24.885943 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:24.885455 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl" podUID="63d87167-397f-476b-837f-e40c8f14a994" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 30 21:52:25.121942 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:25.121910 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:25.879090 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:25.879051 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:25.883633 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:25.883601 6720 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 30 21:52:25.883777 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:25.883650 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl" podUID="63d87167-397f-476b-837f-e40c8f14a994" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 30 21:52:26.142736 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:26.142638 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:26.744104 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:26.744066 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:26.883488 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:26.883458 6720 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 30 21:52:26.883640 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:26.883516 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl" podUID="63d87167-397f-476b-837f-e40c8f14a994" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 30 21:52:27.572279 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:27.572240 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:27.883720 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:27.883684 6720 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 30 21:52:27.883906 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:27.883769 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl" podUID="63d87167-397f-476b-837f-e40c8f14a994" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 30 21:52:28.085789 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:28.084505 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-ceb36a/e2e-conflict-b-dfac3fd4-kserve-854d9d66d8-mfz6k"] Oct 30 21:52:28.088710 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:28.088689 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ceb36a/e2e-conflict-b-dfac3fd4-kserve-854d9d66d8-mfz6k" Oct 30 21:52:28.098143 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:28.098109 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-ceb36a\"/\"e2e-conflict-b-dfac3fd4-kserve-self-signed-certs\"" Oct 30 21:52:28.102409 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:28.100163 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-ceb36a/e2e-conflict-b-dfac3fd4-kserve-854d9d66d8-mfz6k"] Oct 30 21:52:28.185649 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:28.185562 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:28.233586 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:28.233467 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b67a39d-9f99-4d75-bd3b-2527181303c7-home\") pod \"e2e-conflict-b-dfac3fd4-kserve-854d9d66d8-mfz6k\" (UID: \"4b67a39d-9f99-4d75-bd3b-2527181303c7\") " pod="e2e-models-e2e-worker-w3-ceb36a/e2e-conflict-b-dfac3fd4-kserve-854d9d66d8-mfz6k" Oct 30 21:52:28.233586 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:28.233511 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b67a39d-9f99-4d75-bd3b-2527181303c7-tls-certs\") pod \"e2e-conflict-b-dfac3fd4-kserve-854d9d66d8-mfz6k\" (UID: \"4b67a39d-9f99-4d75-bd3b-2527181303c7\") " pod="e2e-models-e2e-worker-w3-ceb36a/e2e-conflict-b-dfac3fd4-kserve-854d9d66d8-mfz6k" Oct 30 21:52:28.233586 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:28.233553 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b67a39d-9f99-4d75-bd3b-2527181303c7-model-cache\") pod \"e2e-conflict-b-dfac3fd4-kserve-854d9d66d8-mfz6k\" (UID: \"4b67a39d-9f99-4d75-bd3b-2527181303c7\") " pod="e2e-models-e2e-worker-w3-ceb36a/e2e-conflict-b-dfac3fd4-kserve-854d9d66d8-mfz6k" Oct 30 21:52:28.233835 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:28.233663 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b67a39d-9f99-4d75-bd3b-2527181303c7-tmp-dir\") pod \"e2e-conflict-b-dfac3fd4-kserve-854d9d66d8-mfz6k\" (UID: \"4b67a39d-9f99-4d75-bd3b-2527181303c7\") " pod="e2e-models-e2e-worker-w3-ceb36a/e2e-conflict-b-dfac3fd4-kserve-854d9d66d8-mfz6k" Oct 30 21:52:28.233835 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:28.233740 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b67a39d-9f99-4d75-bd3b-2527181303c7-dshm\") pod \"e2e-conflict-b-dfac3fd4-kserve-854d9d66d8-mfz6k\" (UID: \"4b67a39d-9f99-4d75-bd3b-2527181303c7\") " pod="e2e-models-e2e-worker-w3-ceb36a/e2e-conflict-b-dfac3fd4-kserve-854d9d66d8-mfz6k" Oct 30 21:52:28.233835 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:28.233809 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnj6d\" (UniqueName: \"kubernetes.io/projected/4b67a39d-9f99-4d75-bd3b-2527181303c7-kube-api-access-fnj6d\") pod \"e2e-conflict-b-dfac3fd4-kserve-854d9d66d8-mfz6k\" (UID: \"4b67a39d-9f99-4d75-bd3b-2527181303c7\") " pod="e2e-models-e2e-worker-w3-ceb36a/e2e-conflict-b-dfac3fd4-kserve-854d9d66d8-mfz6k" Oct 30 21:52:28.335451 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:28.335377 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b67a39d-9f99-4d75-bd3b-2527181303c7-home\") pod \"e2e-conflict-b-dfac3fd4-kserve-854d9d66d8-mfz6k\" (UID: \"4b67a39d-9f99-4d75-bd3b-2527181303c7\") " pod="e2e-models-e2e-worker-w3-ceb36a/e2e-conflict-b-dfac3fd4-kserve-854d9d66d8-mfz6k" Oct 30 21:52:28.335706 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:28.335688 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b67a39d-9f99-4d75-bd3b-2527181303c7-tls-certs\") pod \"e2e-conflict-b-dfac3fd4-kserve-854d9d66d8-mfz6k\" (UID: \"4b67a39d-9f99-4d75-bd3b-2527181303c7\") " pod="e2e-models-e2e-worker-w3-ceb36a/e2e-conflict-b-dfac3fd4-kserve-854d9d66d8-mfz6k" Oct 30 21:52:28.335838 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:28.335824 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b67a39d-9f99-4d75-bd3b-2527181303c7-model-cache\") pod \"e2e-conflict-b-dfac3fd4-kserve-854d9d66d8-mfz6k\" (UID: \"4b67a39d-9f99-4d75-bd3b-2527181303c7\") " pod="e2e-models-e2e-worker-w3-ceb36a/e2e-conflict-b-dfac3fd4-kserve-854d9d66d8-mfz6k" Oct 30 21:52:28.336048 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:28.336031 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b67a39d-9f99-4d75-bd3b-2527181303c7-tmp-dir\") pod \"e2e-conflict-b-dfac3fd4-kserve-854d9d66d8-mfz6k\" (UID: \"4b67a39d-9f99-4d75-bd3b-2527181303c7\") " pod="e2e-models-e2e-worker-w3-ceb36a/e2e-conflict-b-dfac3fd4-kserve-854d9d66d8-mfz6k" Oct 30 21:52:28.336202 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:28.335772 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b67a39d-9f99-4d75-bd3b-2527181303c7-home\") pod \"e2e-conflict-b-dfac3fd4-kserve-854d9d66d8-mfz6k\" (UID: \"4b67a39d-9f99-4d75-bd3b-2527181303c7\") " pod="e2e-models-e2e-worker-w3-ceb36a/e2e-conflict-b-dfac3fd4-kserve-854d9d66d8-mfz6k" Oct 30 21:52:28.336268 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:28.336181 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b67a39d-9f99-4d75-bd3b-2527181303c7-dshm\") pod \"e2e-conflict-b-dfac3fd4-kserve-854d9d66d8-mfz6k\" (UID: \"4b67a39d-9f99-4d75-bd3b-2527181303c7\") " pod="e2e-models-e2e-worker-w3-ceb36a/e2e-conflict-b-dfac3fd4-kserve-854d9d66d8-mfz6k" Oct 30 21:52:28.336378 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:28.336355 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnj6d\" (UniqueName: \"kubernetes.io/projected/4b67a39d-9f99-4d75-bd3b-2527181303c7-kube-api-access-fnj6d\") pod \"e2e-conflict-b-dfac3fd4-kserve-854d9d66d8-mfz6k\" (UID: \"4b67a39d-9f99-4d75-bd3b-2527181303c7\") " pod="e2e-models-e2e-worker-w3-ceb36a/e2e-conflict-b-dfac3fd4-kserve-854d9d66d8-mfz6k" Oct 30 21:52:28.336478 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:28.336437 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b67a39d-9f99-4d75-bd3b-2527181303c7-tmp-dir\") pod \"e2e-conflict-b-dfac3fd4-kserve-854d9d66d8-mfz6k\" (UID: \"4b67a39d-9f99-4d75-bd3b-2527181303c7\") " pod="e2e-models-e2e-worker-w3-ceb36a/e2e-conflict-b-dfac3fd4-kserve-854d9d66d8-mfz6k" Oct 30 21:52:28.336534 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:28.336363 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b67a39d-9f99-4d75-bd3b-2527181303c7-model-cache\") pod \"e2e-conflict-b-dfac3fd4-kserve-854d9d66d8-mfz6k\" (UID: \"4b67a39d-9f99-4d75-bd3b-2527181303c7\") " pod="e2e-models-e2e-worker-w3-ceb36a/e2e-conflict-b-dfac3fd4-kserve-854d9d66d8-mfz6k" Oct 30 21:52:28.338574 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:28.338548 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b67a39d-9f99-4d75-bd3b-2527181303c7-tls-certs\") pod \"e2e-conflict-b-dfac3fd4-kserve-854d9d66d8-mfz6k\" (UID: \"4b67a39d-9f99-4d75-bd3b-2527181303c7\") " pod="e2e-models-e2e-worker-w3-ceb36a/e2e-conflict-b-dfac3fd4-kserve-854d9d66d8-mfz6k" Oct 30 21:52:28.339233 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:28.339213 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b67a39d-9f99-4d75-bd3b-2527181303c7-dshm\") pod \"e2e-conflict-b-dfac3fd4-kserve-854d9d66d8-mfz6k\" (UID: \"4b67a39d-9f99-4d75-bd3b-2527181303c7\") " pod="e2e-models-e2e-worker-w3-ceb36a/e2e-conflict-b-dfac3fd4-kserve-854d9d66d8-mfz6k" Oct 30 21:52:28.374177 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:28.374101 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnj6d\" (UniqueName: \"kubernetes.io/projected/4b67a39d-9f99-4d75-bd3b-2527181303c7-kube-api-access-fnj6d\") pod \"e2e-conflict-b-dfac3fd4-kserve-854d9d66d8-mfz6k\" (UID: \"4b67a39d-9f99-4d75-bd3b-2527181303c7\") " pod="e2e-models-e2e-worker-w3-ceb36a/e2e-conflict-b-dfac3fd4-kserve-854d9d66d8-mfz6k" Oct 30 21:52:28.411955 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:28.411913 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ceb36a/e2e-conflict-b-dfac3fd4-kserve-854d9d66d8-mfz6k" Oct 30 21:52:28.490786 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:28.490710 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:28.723862 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:28.723835 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-ceb36a/e2e-conflict-b-dfac3fd4-kserve-854d9d66d8-mfz6k"] Oct 30 21:52:28.726188 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:52:28.726161 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4b67a39d_9f99_4d75_bd3b_2527181303c7.slice/crio-85f8d1c51576dda50fd1d8c1399be530753c959e43761b5fa95b4f2265f1b77f WatchSource:0}: Error finding container 85f8d1c51576dda50fd1d8c1399be530753c959e43761b5fa95b4f2265f1b77f: Status 404 returned error can't find the container with id 85f8d1c51576dda50fd1d8c1399be530753c959e43761b5fa95b4f2265f1b77f Oct 30 21:52:28.883939 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:28.883897 6720 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Oct 30 21:52:28.884117 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:28.883961 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl" podUID="63d87167-397f-476b-837f-e40c8f14a994" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Oct 30 21:52:29.403411 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:29.402450 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:29.442118 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:29.442084 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ceb36a/e2e-conflict-b-dfac3fd4-kserve-854d9d66d8-mfz6k" event={"ID":"4b67a39d-9f99-4d75-bd3b-2527181303c7","Type":"ContainerStarted","Data":"2e05ef665738e4c943a7c413f892e85c838bc426fec2bf694b625282c5fbb8b2"} Oct 30 21:52:29.442292 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:29.442129 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ceb36a/e2e-conflict-b-dfac3fd4-kserve-854d9d66d8-mfz6k" event={"ID":"4b67a39d-9f99-4d75-bd3b-2527181303c7","Type":"ContainerStarted","Data":"85f8d1c51576dda50fd1d8c1399be530753c959e43761b5fa95b4f2265f1b77f"} Oct 30 21:52:29.468681 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:29.468625 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-ceb36a/e2e-conflict-b-dfac3fd4-kserve-854d9d66d8-mfz6k" podStartSLOduration=1.468609674 podStartE2EDuration="1.468609674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:29.465690064 +0000 UTC m=+715.636655246" watchObservedRunningTime="2026-10-30 21:52:29.468609674 +0000 UTC m=+715.639574855" Oct 30 21:52:30.016494 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:30.016455 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl" Oct 30 21:52:30.017135 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:30.017101 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl" Oct 30 21:52:30.018030 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:30.018009 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl" Oct 30 21:52:30.329480 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:30.329366 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:30.417000 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:30.416963 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ceb36a/e2e-conflict-b-dfac3fd4-kserve-854d9d66d8-mfz6k" Oct 30 21:52:31.031406 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:31.031255 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:32.066411 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:32.065840 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:33.017634 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:33.017592 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:33.790268 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:33.790236 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:34.571857 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:34.571826 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:35.375985 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:35.375945 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:35.929637 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:35.929597 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-939a16/distinct-w4-939a16-kserve-7f88f68c56-cngtk"] Oct 30 21:52:35.933753 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:35.933727 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-939a16/distinct-w4-939a16-kserve-7f88f68c56-cngtk" Oct 30 21:52:35.943631 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:35.943590 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-939a16\"/\"distinct-w4-939a16-kserve-self-signed-certs\"" Oct 30 21:52:35.961884 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:35.961845 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlhzx\" (UniqueName: \"kubernetes.io/projected/85316c69-7ebd-4fb6-9195-44d1fba2dfbc-kube-api-access-hlhzx\") pod \"distinct-w4-939a16-kserve-7f88f68c56-cngtk\" (UID: \"85316c69-7ebd-4fb6-9195-44d1fba2dfbc\") " pod="e2e-models-e2e-worker-w4-939a16/distinct-w4-939a16-kserve-7f88f68c56-cngtk" Oct 30 21:52:35.962810 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:35.962772 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85316c69-7ebd-4fb6-9195-44d1fba2dfbc-model-cache\") pod \"distinct-w4-939a16-kserve-7f88f68c56-cngtk\" (UID: \"85316c69-7ebd-4fb6-9195-44d1fba2dfbc\") " pod="e2e-models-e2e-worker-w4-939a16/distinct-w4-939a16-kserve-7f88f68c56-cngtk" Oct 30 21:52:35.963135 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:35.963108 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85316c69-7ebd-4fb6-9195-44d1fba2dfbc-tls-certs\") pod \"distinct-w4-939a16-kserve-7f88f68c56-cngtk\" (UID: \"85316c69-7ebd-4fb6-9195-44d1fba2dfbc\") " pod="e2e-models-e2e-worker-w4-939a16/distinct-w4-939a16-kserve-7f88f68c56-cngtk" Oct 30 21:52:35.969620 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:35.969598 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-939a16/distinct-w4-939a16-kserve-7f88f68c56-cngtk"] Oct 30 21:52:35.971465 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:35.971439 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85316c69-7ebd-4fb6-9195-44d1fba2dfbc-home\") pod \"distinct-w4-939a16-kserve-7f88f68c56-cngtk\" (UID: \"85316c69-7ebd-4fb6-9195-44d1fba2dfbc\") " pod="e2e-models-e2e-worker-w4-939a16/distinct-w4-939a16-kserve-7f88f68c56-cngtk" Oct 30 21:52:35.971579 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:35.971529 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85316c69-7ebd-4fb6-9195-44d1fba2dfbc-dshm\") pod \"distinct-w4-939a16-kserve-7f88f68c56-cngtk\" (UID: \"85316c69-7ebd-4fb6-9195-44d1fba2dfbc\") " pod="e2e-models-e2e-worker-w4-939a16/distinct-w4-939a16-kserve-7f88f68c56-cngtk" Oct 30 21:52:35.971579 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:35.971560 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85316c69-7ebd-4fb6-9195-44d1fba2dfbc-tmp-dir\") pod \"distinct-w4-939a16-kserve-7f88f68c56-cngtk\" (UID: \"85316c69-7ebd-4fb6-9195-44d1fba2dfbc\") " pod="e2e-models-e2e-worker-w4-939a16/distinct-w4-939a16-kserve-7f88f68c56-cngtk" Oct 30 21:52:36.046718 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:36.046681 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:36.073097 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:36.073054 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85316c69-7ebd-4fb6-9195-44d1fba2dfbc-tls-certs\") pod \"distinct-w4-939a16-kserve-7f88f68c56-cngtk\" (UID: \"85316c69-7ebd-4fb6-9195-44d1fba2dfbc\") " pod="e2e-models-e2e-worker-w4-939a16/distinct-w4-939a16-kserve-7f88f68c56-cngtk" Oct 30 21:52:36.073286 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:36.073121 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85316c69-7ebd-4fb6-9195-44d1fba2dfbc-home\") pod \"distinct-w4-939a16-kserve-7f88f68c56-cngtk\" (UID: \"85316c69-7ebd-4fb6-9195-44d1fba2dfbc\") " pod="e2e-models-e2e-worker-w4-939a16/distinct-w4-939a16-kserve-7f88f68c56-cngtk" Oct 30 21:52:36.073286 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:36.073209 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85316c69-7ebd-4fb6-9195-44d1fba2dfbc-dshm\") pod \"distinct-w4-939a16-kserve-7f88f68c56-cngtk\" (UID: \"85316c69-7ebd-4fb6-9195-44d1fba2dfbc\") " pod="e2e-models-e2e-worker-w4-939a16/distinct-w4-939a16-kserve-7f88f68c56-cngtk" Oct 30 21:52:36.073286 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:36.073236 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85316c69-7ebd-4fb6-9195-44d1fba2dfbc-tmp-dir\") pod \"distinct-w4-939a16-kserve-7f88f68c56-cngtk\" (UID: \"85316c69-7ebd-4fb6-9195-44d1fba2dfbc\") " pod="e2e-models-e2e-worker-w4-939a16/distinct-w4-939a16-kserve-7f88f68c56-cngtk" Oct 30 21:52:36.073286 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:36.073280 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlhzx\" (UniqueName: \"kubernetes.io/projected/85316c69-7ebd-4fb6-9195-44d1fba2dfbc-kube-api-access-hlhzx\") pod \"distinct-w4-939a16-kserve-7f88f68c56-cngtk\" (UID: \"85316c69-7ebd-4fb6-9195-44d1fba2dfbc\") " pod="e2e-models-e2e-worker-w4-939a16/distinct-w4-939a16-kserve-7f88f68c56-cngtk" Oct 30 21:52:36.073560 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:36.073408 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85316c69-7ebd-4fb6-9195-44d1fba2dfbc-model-cache\") pod \"distinct-w4-939a16-kserve-7f88f68c56-cngtk\" (UID: \"85316c69-7ebd-4fb6-9195-44d1fba2dfbc\") " pod="e2e-models-e2e-worker-w4-939a16/distinct-w4-939a16-kserve-7f88f68c56-cngtk" Oct 30 21:52:36.073888 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:36.073862 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85316c69-7ebd-4fb6-9195-44d1fba2dfbc-tmp-dir\") pod \"distinct-w4-939a16-kserve-7f88f68c56-cngtk\" (UID: \"85316c69-7ebd-4fb6-9195-44d1fba2dfbc\") " pod="e2e-models-e2e-worker-w4-939a16/distinct-w4-939a16-kserve-7f88f68c56-cngtk" Oct 30 21:52:36.073981 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:36.073914 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85316c69-7ebd-4fb6-9195-44d1fba2dfbc-model-cache\") pod \"distinct-w4-939a16-kserve-7f88f68c56-cngtk\" (UID: \"85316c69-7ebd-4fb6-9195-44d1fba2dfbc\") " pod="e2e-models-e2e-worker-w4-939a16/distinct-w4-939a16-kserve-7f88f68c56-cngtk" Oct 30 21:52:36.074898 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:36.074871 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85316c69-7ebd-4fb6-9195-44d1fba2dfbc-home\") pod \"distinct-w4-939a16-kserve-7f88f68c56-cngtk\" (UID: \"85316c69-7ebd-4fb6-9195-44d1fba2dfbc\") " pod="e2e-models-e2e-worker-w4-939a16/distinct-w4-939a16-kserve-7f88f68c56-cngtk" Oct 30 21:52:36.077418 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:36.077372 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85316c69-7ebd-4fb6-9195-44d1fba2dfbc-dshm\") pod \"distinct-w4-939a16-kserve-7f88f68c56-cngtk\" (UID: \"85316c69-7ebd-4fb6-9195-44d1fba2dfbc\") " pod="e2e-models-e2e-worker-w4-939a16/distinct-w4-939a16-kserve-7f88f68c56-cngtk" Oct 30 21:52:36.077583 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:36.077454 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85316c69-7ebd-4fb6-9195-44d1fba2dfbc-tls-certs\") pod \"distinct-w4-939a16-kserve-7f88f68c56-cngtk\" (UID: \"85316c69-7ebd-4fb6-9195-44d1fba2dfbc\") " pod="e2e-models-e2e-worker-w4-939a16/distinct-w4-939a16-kserve-7f88f68c56-cngtk" Oct 30 21:52:36.088168 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:36.088138 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlhzx\" (UniqueName: \"kubernetes.io/projected/85316c69-7ebd-4fb6-9195-44d1fba2dfbc-kube-api-access-hlhzx\") pod \"distinct-w4-939a16-kserve-7f88f68c56-cngtk\" (UID: \"85316c69-7ebd-4fb6-9195-44d1fba2dfbc\") " pod="e2e-models-e2e-worker-w4-939a16/distinct-w4-939a16-kserve-7f88f68c56-cngtk" Oct 30 21:52:36.233674 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:36.233596 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:36.248570 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:36.248538 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-939a16/distinct-w4-939a16-kserve-7f88f68c56-cngtk" Oct 30 21:52:36.526540 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:36.526512 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-939a16/distinct-w4-939a16-kserve-7f88f68c56-cngtk"] Oct 30 21:52:36.528250 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:52:36.528225 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod85316c69_7ebd_4fb6_9195_44d1fba2dfbc.slice/crio-5f9a70dd029eacdb2af4836cddeabe1e24a5f218937d677e0d3e68a5261989de WatchSource:0}: Error finding container 5f9a70dd029eacdb2af4836cddeabe1e24a5f218937d677e0d3e68a5261989de: Status 404 returned error can't find the container with id 5f9a70dd029eacdb2af4836cddeabe1e24a5f218937d677e0d3e68a5261989de Oct 30 21:52:36.939161 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:36.939129 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:37.523007 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:37.522975 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-939a16/distinct-w4-939a16-kserve-7f88f68c56-cngtk" event={"ID":"85316c69-7ebd-4fb6-9195-44d1fba2dfbc","Type":"ContainerStarted","Data":"171820a5d8cdf5c032aecaaa7c783a831940885f2d5065de1c2159b3ae8adad2"} Oct 30 21:52:37.523163 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:37.523021 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-939a16/distinct-w4-939a16-kserve-7f88f68c56-cngtk" event={"ID":"85316c69-7ebd-4fb6-9195-44d1fba2dfbc","Type":"ContainerStarted","Data":"5f9a70dd029eacdb2af4836cddeabe1e24a5f218937d677e0d3e68a5261989de"} Oct 30 21:52:37.542704 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:37.542652 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-939a16/distinct-w4-939a16-kserve-7f88f68c56-cngtk" podStartSLOduration=2.542638675 podStartE2EDuration="2.542638675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:37.542554008 +0000 UTC m=+723.713519192" watchObservedRunningTime="2026-10-30 21:52:37.542638675 +0000 UTC m=+723.713603857" Oct 30 21:52:37.718132 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:37.718100 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:38.249163 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:38.249134 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-939a16/distinct-w4-939a16-kserve-7f88f68c56-cngtk" Oct 30 21:52:38.406193 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:38.406148 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:38.417115 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:38.417089 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-ceb36a/e2e-conflict-b-dfac3fd4-kserve-854d9d66d8-mfz6k" Oct 30 21:52:38.429879 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:38.429853 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-ceb36a/e2e-conflict-b-dfac3fd4-kserve-854d9d66d8-mfz6k" Oct 30 21:52:38.540967 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:38.540890 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ceb36a/e2e-conflict-b-dfac3fd4-kserve-854d9d66d8-mfz6k" Oct 30 21:52:39.115433 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:39.115399 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:39.740896 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:39.740866 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:40.405302 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:52:40.405255 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 21:52:40.411708 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:40.411685 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:41.082131 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:41.082095 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-1529dc/distinct-2-w0-1529dc-kserve-7788894c6-7mbrr"] Oct 30 21:52:41.086220 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:41.086194 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1529dc/distinct-2-w0-1529dc-kserve-7788894c6-7mbrr" Oct 30 21:52:41.099893 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:41.099871 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-1529dc\"/\"distinct-2-w0-1529dc-kserve-self-signed-certs\"" Oct 30 21:52:41.107449 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:41.107372 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1529dc/distinct-2-w0-1529dc-kserve-7788894c6-7mbrr"] Oct 30 21:52:41.128441 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:41.128378 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09662f19-1ef9-406c-85ef-e402505875b4-tls-certs\") pod \"distinct-2-w0-1529dc-kserve-7788894c6-7mbrr\" (UID: \"09662f19-1ef9-406c-85ef-e402505875b4\") " pod="e2e-models-e2e-worker-w0-1529dc/distinct-2-w0-1529dc-kserve-7788894c6-7mbrr" Oct 30 21:52:41.128570 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:41.128452 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4h65\" (UniqueName: \"kubernetes.io/projected/09662f19-1ef9-406c-85ef-e402505875b4-kube-api-access-w4h65\") pod \"distinct-2-w0-1529dc-kserve-7788894c6-7mbrr\" (UID: \"09662f19-1ef9-406c-85ef-e402505875b4\") " pod="e2e-models-e2e-worker-w0-1529dc/distinct-2-w0-1529dc-kserve-7788894c6-7mbrr" Oct 30 21:52:41.128636 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:41.128560 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09662f19-1ef9-406c-85ef-e402505875b4-tmp-dir\") pod \"distinct-2-w0-1529dc-kserve-7788894c6-7mbrr\" (UID: \"09662f19-1ef9-406c-85ef-e402505875b4\") " pod="e2e-models-e2e-worker-w0-1529dc/distinct-2-w0-1529dc-kserve-7788894c6-7mbrr" Oct 30 21:52:41.128691 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:41.128630 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09662f19-1ef9-406c-85ef-e402505875b4-model-cache\") pod \"distinct-2-w0-1529dc-kserve-7788894c6-7mbrr\" (UID: \"09662f19-1ef9-406c-85ef-e402505875b4\") " pod="e2e-models-e2e-worker-w0-1529dc/distinct-2-w0-1529dc-kserve-7788894c6-7mbrr" Oct 30 21:52:41.128781 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:41.128759 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09662f19-1ef9-406c-85ef-e402505875b4-dshm\") pod \"distinct-2-w0-1529dc-kserve-7788894c6-7mbrr\" (UID: \"09662f19-1ef9-406c-85ef-e402505875b4\") " pod="e2e-models-e2e-worker-w0-1529dc/distinct-2-w0-1529dc-kserve-7788894c6-7mbrr" Oct 30 21:52:41.128848 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:41.128831 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09662f19-1ef9-406c-85ef-e402505875b4-home\") pod \"distinct-2-w0-1529dc-kserve-7788894c6-7mbrr\" (UID: \"09662f19-1ef9-406c-85ef-e402505875b4\") " pod="e2e-models-e2e-worker-w0-1529dc/distinct-2-w0-1529dc-kserve-7788894c6-7mbrr" Oct 30 21:52:41.142165 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:41.142143 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:41.229711 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:41.229680 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09662f19-1ef9-406c-85ef-e402505875b4-dshm\") pod \"distinct-2-w0-1529dc-kserve-7788894c6-7mbrr\" (UID: \"09662f19-1ef9-406c-85ef-e402505875b4\") " pod="e2e-models-e2e-worker-w0-1529dc/distinct-2-w0-1529dc-kserve-7788894c6-7mbrr" Oct 30 21:52:41.229898 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:41.229734 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09662f19-1ef9-406c-85ef-e402505875b4-home\") pod \"distinct-2-w0-1529dc-kserve-7788894c6-7mbrr\" (UID: \"09662f19-1ef9-406c-85ef-e402505875b4\") " pod="e2e-models-e2e-worker-w0-1529dc/distinct-2-w0-1529dc-kserve-7788894c6-7mbrr" Oct 30 21:52:41.229898 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:41.229803 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09662f19-1ef9-406c-85ef-e402505875b4-tls-certs\") pod \"distinct-2-w0-1529dc-kserve-7788894c6-7mbrr\" (UID: \"09662f19-1ef9-406c-85ef-e402505875b4\") " pod="e2e-models-e2e-worker-w0-1529dc/distinct-2-w0-1529dc-kserve-7788894c6-7mbrr" Oct 30 21:52:41.229898 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:41.229844 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4h65\" (UniqueName: \"kubernetes.io/projected/09662f19-1ef9-406c-85ef-e402505875b4-kube-api-access-w4h65\") pod \"distinct-2-w0-1529dc-kserve-7788894c6-7mbrr\" (UID: \"09662f19-1ef9-406c-85ef-e402505875b4\") " pod="e2e-models-e2e-worker-w0-1529dc/distinct-2-w0-1529dc-kserve-7788894c6-7mbrr" Oct 30 21:52:41.229898 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:41.229871 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09662f19-1ef9-406c-85ef-e402505875b4-tmp-dir\") pod \"distinct-2-w0-1529dc-kserve-7788894c6-7mbrr\" (UID: \"09662f19-1ef9-406c-85ef-e402505875b4\") " pod="e2e-models-e2e-worker-w0-1529dc/distinct-2-w0-1529dc-kserve-7788894c6-7mbrr" Oct 30 21:52:41.230147 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:41.229927 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09662f19-1ef9-406c-85ef-e402505875b4-model-cache\") pod \"distinct-2-w0-1529dc-kserve-7788894c6-7mbrr\" (UID: \"09662f19-1ef9-406c-85ef-e402505875b4\") " pod="e2e-models-e2e-worker-w0-1529dc/distinct-2-w0-1529dc-kserve-7788894c6-7mbrr" Oct 30 21:52:41.230252 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:41.230226 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09662f19-1ef9-406c-85ef-e402505875b4-home\") pod \"distinct-2-w0-1529dc-kserve-7788894c6-7mbrr\" (UID: \"09662f19-1ef9-406c-85ef-e402505875b4\") " pod="e2e-models-e2e-worker-w0-1529dc/distinct-2-w0-1529dc-kserve-7788894c6-7mbrr" Oct 30 21:52:41.230319 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:41.230268 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09662f19-1ef9-406c-85ef-e402505875b4-model-cache\") pod \"distinct-2-w0-1529dc-kserve-7788894c6-7mbrr\" (UID: \"09662f19-1ef9-406c-85ef-e402505875b4\") " pod="e2e-models-e2e-worker-w0-1529dc/distinct-2-w0-1529dc-kserve-7788894c6-7mbrr" Oct 30 21:52:41.230552 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:41.230528 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09662f19-1ef9-406c-85ef-e402505875b4-tmp-dir\") pod \"distinct-2-w0-1529dc-kserve-7788894c6-7mbrr\" (UID: \"09662f19-1ef9-406c-85ef-e402505875b4\") " pod="e2e-models-e2e-worker-w0-1529dc/distinct-2-w0-1529dc-kserve-7788894c6-7mbrr" Oct 30 21:52:41.232287 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:41.232261 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09662f19-1ef9-406c-85ef-e402505875b4-dshm\") pod \"distinct-2-w0-1529dc-kserve-7788894c6-7mbrr\" (UID: \"09662f19-1ef9-406c-85ef-e402505875b4\") " pod="e2e-models-e2e-worker-w0-1529dc/distinct-2-w0-1529dc-kserve-7788894c6-7mbrr" Oct 30 21:52:41.233499 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:41.233474 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09662f19-1ef9-406c-85ef-e402505875b4-tls-certs\") pod \"distinct-2-w0-1529dc-kserve-7788894c6-7mbrr\" (UID: \"09662f19-1ef9-406c-85ef-e402505875b4\") " pod="e2e-models-e2e-worker-w0-1529dc/distinct-2-w0-1529dc-kserve-7788894c6-7mbrr" Oct 30 21:52:41.245337 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:41.245310 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4h65\" (UniqueName: \"kubernetes.io/projected/09662f19-1ef9-406c-85ef-e402505875b4-kube-api-access-w4h65\") pod \"distinct-2-w0-1529dc-kserve-7788894c6-7mbrr\" (UID: \"09662f19-1ef9-406c-85ef-e402505875b4\") " pod="e2e-models-e2e-worker-w0-1529dc/distinct-2-w0-1529dc-kserve-7788894c6-7mbrr" Oct 30 21:52:41.303036 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:41.303005 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:41.400235 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:41.400206 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1529dc/distinct-2-w0-1529dc-kserve-7788894c6-7mbrr" Oct 30 21:52:41.470420 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:41.465980 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:41.730942 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:41.730803 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-1529dc/distinct-2-w0-1529dc-kserve-7788894c6-7mbrr"] Oct 30 21:52:41.734370 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:52:41.734325 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod09662f19_1ef9_406c_85ef_e402505875b4.slice/crio-0bae8776c1b7ba368eb85dc7af161beeaf62c542a322e7d3fc31cf954db9bd51 WatchSource:0}: Error finding container 0bae8776c1b7ba368eb85dc7af161beeaf62c542a322e7d3fc31cf954db9bd51: Status 404 returned error can't find the container with id 0bae8776c1b7ba368eb85dc7af161beeaf62c542a322e7d3fc31cf954db9bd51 Oct 30 21:52:42.271404 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:42.270127 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:42.561223 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:42.561125 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1529dc/distinct-2-w0-1529dc-kserve-7788894c6-7mbrr" event={"ID":"09662f19-1ef9-406c-85ef-e402505875b4","Type":"ContainerStarted","Data":"d3ec372bef2802692497034c99efdf7ae1b9b48ee8f973237376d07ff6f04247"} Oct 30 21:52:42.561223 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:42.561178 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1529dc/distinct-2-w0-1529dc-kserve-7788894c6-7mbrr" event={"ID":"09662f19-1ef9-406c-85ef-e402505875b4","Type":"ContainerStarted","Data":"0bae8776c1b7ba368eb85dc7af161beeaf62c542a322e7d3fc31cf954db9bd51"} Oct 30 21:52:42.583895 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:42.583834 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-1529dc/distinct-2-w0-1529dc-kserve-7788894c6-7mbrr" podStartSLOduration=1.583815259 podStartE2EDuration="1.583815259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:42.581707046 +0000 UTC m=+728.752672225" watchObservedRunningTime="2026-10-30 21:52:42.583815259 +0000 UTC m=+728.754780442" Oct 30 21:52:43.314257 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:43.314196 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:43.401237 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:43.401102 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1529dc/distinct-2-w0-1529dc-kserve-7788894c6-7mbrr" Oct 30 21:52:44.336898 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:44.336850 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:45.323227 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:45.323195 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:46.022832 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:46.022791 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:46.248848 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:46.248816 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-939a16/distinct-w4-939a16-kserve-7f88f68c56-cngtk" Oct 30 21:52:46.262131 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:46.262107 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-939a16/distinct-w4-939a16-kserve-7f88f68c56-cngtk" Oct 30 21:52:46.619768 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:46.619739 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-939a16/distinct-w4-939a16-kserve-7f88f68c56-cngtk" Oct 30 21:52:46.716273 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:46.716243 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:47.540174 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:47.540136 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:48.247303 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:48.247270 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:49.009843 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:49.009803 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:49.220343 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:49.220302 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-ceb36a/e2e-conflict-b-dfac3fd4-kserve-854d9d66d8-mfz6k"] Oct 30 21:52:49.297857 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:49.297768 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:49.720957 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:49.720922 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:50.502080 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:50.502045 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:51.316992 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:51.316961 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:51.400459 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:51.400430 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-1529dc/distinct-2-w0-1529dc-kserve-7788894c6-7mbrr" Oct 30 21:52:51.412902 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:51.412877 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-1529dc/distinct-2-w0-1529dc-kserve-7788894c6-7mbrr" Oct 30 21:52:51.646080 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:51.646051 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1529dc/distinct-2-w0-1529dc-kserve-7788894c6-7mbrr" Oct 30 21:52:52.048116 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:52.048035 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:52.720020 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:52.719981 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:53.660916 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:53.660884 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:54.463165 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:54.463120 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:54.803631 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:54.803538 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-ceb36a/e2e-conflict-a-dfac3fd4-kserve-677cc88846-vf5kf"] Oct 30 21:52:54.896171 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:54.896124 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:55.743292 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:55.743254 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:56.892421 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:56.891951 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:57.057129 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:57.057068 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:57.931330 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:57.931282 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:58.621643 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:58.621611 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:52:59.376891 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:52:59.376859 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:00.143615 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:00.143582 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:00.822113 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:00.822083 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:01.525114 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:01.525084 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:02.233572 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:02.233517 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:02.926047 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:02.926013 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:03.545672 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:03.545639 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:04.218064 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:04.218038 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:04.243796 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:04.243758 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-ceb36a/e2e-conflict-b-dfac3fd4-kserve-854d9d66d8-mfz6k" podUID="4b67a39d-9f99-4d75-bd3b-2527181303c7" containerName="main" containerID="cri-o://2e05ef665738e4c943a7c413f892e85c838bc426fec2bf694b625282c5fbb8b2" gracePeriod=45 Oct 30 21:53:04.518168 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:04.517886 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ceb36a/e2e-conflict-b-dfac3fd4-kserve-854d9d66d8-mfz6k" Oct 30 21:53:04.576244 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:04.576217 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b67a39d-9f99-4d75-bd3b-2527181303c7-home\" (UniqueName: \"kubernetes.io/empty-dir/4b67a39d-9f99-4d75-bd3b-2527181303c7-home\") pod \"4b67a39d-9f99-4d75-bd3b-2527181303c7\" (UID: \"4b67a39d-9f99-4d75-bd3b-2527181303c7\") " Oct 30 21:53:04.576716 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:04.576261 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b67a39d-9f99-4d75-bd3b-2527181303c7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b67a39d-9f99-4d75-bd3b-2527181303c7-dshm\") pod \"4b67a39d-9f99-4d75-bd3b-2527181303c7\" (UID: \"4b67a39d-9f99-4d75-bd3b-2527181303c7\") " Oct 30 21:53:04.576716 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:04.576308 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b67a39d-9f99-4d75-bd3b-2527181303c7-tls-certs\" (UniqueName: \"kubernetes.io/secret/4b67a39d-9f99-4d75-bd3b-2527181303c7-tls-certs\") pod \"4b67a39d-9f99-4d75-bd3b-2527181303c7\" (UID: \"4b67a39d-9f99-4d75-bd3b-2527181303c7\") " Oct 30 21:53:04.576716 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:04.576415 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b67a39d-9f99-4d75-bd3b-2527181303c7-kube-api-access-fnj6d\" (UniqueName: \"kubernetes.io/projected/4b67a39d-9f99-4d75-bd3b-2527181303c7-kube-api-access-fnj6d\") pod \"4b67a39d-9f99-4d75-bd3b-2527181303c7\" (UID: \"4b67a39d-9f99-4d75-bd3b-2527181303c7\") " Oct 30 21:53:04.576716 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:04.576537 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b67a39d-9f99-4d75-bd3b-2527181303c7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b67a39d-9f99-4d75-bd3b-2527181303c7-tmp-dir\") pod \"4b67a39d-9f99-4d75-bd3b-2527181303c7\" (UID: \"4b67a39d-9f99-4d75-bd3b-2527181303c7\") " Oct 30 21:53:04.576716 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:04.576551 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b67a39d-9f99-4d75-bd3b-2527181303c7-home" pod "4b67a39d-9f99-4d75-bd3b-2527181303c7" (UID: "4b67a39d-9f99-4d75-bd3b-2527181303c7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:53:04.576716 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:04.576620 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b67a39d-9f99-4d75-bd3b-2527181303c7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b67a39d-9f99-4d75-bd3b-2527181303c7-model-cache\") pod \"4b67a39d-9f99-4d75-bd3b-2527181303c7\" (UID: \"4b67a39d-9f99-4d75-bd3b-2527181303c7\") " Oct 30 21:53:04.577283 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:04.576833 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b67a39d-9f99-4d75-bd3b-2527181303c7-tmp-dir" pod "4b67a39d-9f99-4d75-bd3b-2527181303c7" (UID: "4b67a39d-9f99-4d75-bd3b-2527181303c7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:53:04.577283 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:04.577004 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b67a39d-9f99-4d75-bd3b-2527181303c7-model-cache" pod "4b67a39d-9f99-4d75-bd3b-2527181303c7" (UID: "4b67a39d-9f99-4d75-bd3b-2527181303c7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:53:04.577283 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:04.577084 6720 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b67a39d-9f99-4d75-bd3b-2527181303c7-tmp-dir\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:53:04.577283 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:04.577102 6720 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b67a39d-9f99-4d75-bd3b-2527181303c7-model-cache\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:53:04.577283 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:04.577115 6720 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b67a39d-9f99-4d75-bd3b-2527181303c7-home\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:53:04.578788 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:04.578752 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b67a39d-9f99-4d75-bd3b-2527181303c7-kube-api-access-fnj6d" pod "4b67a39d-9f99-4d75-bd3b-2527181303c7" (UID: "4b67a39d-9f99-4d75-bd3b-2527181303c7"). InnerVolumeSpecName "kube-api-access-fnj6d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:53:04.578907 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:04.578885 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b67a39d-9f99-4d75-bd3b-2527181303c7-tls-certs" pod "4b67a39d-9f99-4d75-bd3b-2527181303c7" (UID: "4b67a39d-9f99-4d75-bd3b-2527181303c7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:53:04.579633 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:04.579613 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b67a39d-9f99-4d75-bd3b-2527181303c7-dshm" pod "4b67a39d-9f99-4d75-bd3b-2527181303c7" (UID: "4b67a39d-9f99-4d75-bd3b-2527181303c7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:53:04.678348 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:04.678311 6720 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b67a39d-9f99-4d75-bd3b-2527181303c7-tls-certs\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:53:04.678348 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:04.678346 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnj6d\" (UniqueName: \"kubernetes.io/projected/4b67a39d-9f99-4d75-bd3b-2527181303c7-kube-api-access-fnj6d\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:53:04.678609 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:04.678362 6720 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b67a39d-9f99-4d75-bd3b-2527181303c7-dshm\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:53:04.741093 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:04.741001 6720 generic.go:356] "Generic (PLEG): container finished" podID="4b67a39d-9f99-4d75-bd3b-2527181303c7" containerID="2e05ef665738e4c943a7c413f892e85c838bc426fec2bf694b625282c5fbb8b2" exitCode=0 Oct 30 21:53:04.741259 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:04.741127 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ceb36a/e2e-conflict-b-dfac3fd4-kserve-854d9d66d8-mfz6k" event={"ID":"4b67a39d-9f99-4d75-bd3b-2527181303c7","Type":"ContainerDied","Data":"2e05ef665738e4c943a7c413f892e85c838bc426fec2bf694b625282c5fbb8b2"} Oct 30 21:53:04.741259 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:04.741166 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ceb36a/e2e-conflict-b-dfac3fd4-kserve-854d9d66d8-mfz6k" event={"ID":"4b67a39d-9f99-4d75-bd3b-2527181303c7","Type":"ContainerDied","Data":"85f8d1c51576dda50fd1d8c1399be530753c959e43761b5fa95b4f2265f1b77f"} Oct 30 21:53:04.741259 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:04.741188 6720 scope.go:122] "RemoveContainer" containerID="2e05ef665738e4c943a7c413f892e85c838bc426fec2bf694b625282c5fbb8b2" Oct 30 21:53:04.741464 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:04.741343 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ceb36a/e2e-conflict-b-dfac3fd4-kserve-854d9d66d8-mfz6k" Oct 30 21:53:04.760139 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:04.760115 6720 scope.go:122] "RemoveContainer" containerID="2e05ef665738e4c943a7c413f892e85c838bc426fec2bf694b625282c5fbb8b2" Oct 30 21:53:04.760480 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:53:04.760457 6720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e05ef665738e4c943a7c413f892e85c838bc426fec2bf694b625282c5fbb8b2\": container with ID starting with 2e05ef665738e4c943a7c413f892e85c838bc426fec2bf694b625282c5fbb8b2 not found: ID does not exist" containerID="2e05ef665738e4c943a7c413f892e85c838bc426fec2bf694b625282c5fbb8b2" Oct 30 21:53:04.760591 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:04.760488 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e05ef665738e4c943a7c413f892e85c838bc426fec2bf694b625282c5fbb8b2"} err="failed to get container status \"2e05ef665738e4c943a7c413f892e85c838bc426fec2bf694b625282c5fbb8b2\": rpc error: code = NotFound desc = could not find container \"2e05ef665738e4c943a7c413f892e85c838bc426fec2bf694b625282c5fbb8b2\": container with ID starting with 2e05ef665738e4c943a7c413f892e85c838bc426fec2bf694b625282c5fbb8b2 not found: ID does not exist" Oct 30 21:53:04.770194 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:04.770163 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-ceb36a/e2e-conflict-b-dfac3fd4-kserve-854d9d66d8-mfz6k"] Oct 30 21:53:04.774716 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:04.774693 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-ceb36a/e2e-conflict-b-dfac3fd4-kserve-854d9d66d8-mfz6k"] Oct 30 21:53:04.940836 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:04.940795 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:05.668443 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:05.668369 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:05.956897 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:05.956817 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b67a39d-9f99-4d75-bd3b-2527181303c7" path="/var/lib/kubelet/pods/4b67a39d-9f99-4d75-bd3b-2527181303c7/volumes" Oct 30 21:53:06.352467 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:06.352153 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:06.851498 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:06.851464 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269"] Oct 30 21:53:06.852188 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:06.852168 6720 cpu_manager.go:414] "containerMap: removing container" podUID="4b67a39d-9f99-4d75-bd3b-2527181303c7" containerName="main" Oct 30 21:53:06.852241 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:06.852192 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b67a39d-9f99-4d75-bd3b-2527181303c7" containerName="main" Oct 30 21:53:06.852378 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:06.852364 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b67a39d-9f99-4d75-bd3b-2527181303c7" containerName="main" Oct 30 21:53:06.857435 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:06.857368 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269" Oct 30 21:53:06.867730 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:06.867703 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-1529dc-openshift-default-dockercfg-zmbwx\"" Oct 30 21:53:06.885307 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:06.885274 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269"] Oct 30 21:53:06.932152 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:06.931448 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01205782-af7b-4eda-894d-3850de540135-workload-socket\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269\" (UID: \"01205782-af7b-4eda-894d-3850de540135\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269" Oct 30 21:53:06.932152 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:06.931521 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01205782-af7b-4eda-894d-3850de540135-credential-socket\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269\" (UID: \"01205782-af7b-4eda-894d-3850de540135\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269" Oct 30 21:53:06.932152 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:06.931563 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01205782-af7b-4eda-894d-3850de540135-istio-podinfo\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269\" (UID: \"01205782-af7b-4eda-894d-3850de540135\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269" Oct 30 21:53:06.932152 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:06.931597 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01205782-af7b-4eda-894d-3850de540135-istio-data\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269\" (UID: \"01205782-af7b-4eda-894d-3850de540135\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269" Oct 30 21:53:06.932152 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:06.931624 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01205782-af7b-4eda-894d-3850de540135-istiod-ca-cert\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269\" (UID: \"01205782-af7b-4eda-894d-3850de540135\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269" Oct 30 21:53:06.932152 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:06.931664 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01205782-af7b-4eda-894d-3850de540135-istio-token\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269\" (UID: \"01205782-af7b-4eda-894d-3850de540135\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269" Oct 30 21:53:06.932152 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:06.931714 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01205782-af7b-4eda-894d-3850de540135-istio-envoy\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269\" (UID: \"01205782-af7b-4eda-894d-3850de540135\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269" Oct 30 21:53:06.932152 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:06.931766 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01205782-af7b-4eda-894d-3850de540135-workload-certs\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269\" (UID: \"01205782-af7b-4eda-894d-3850de540135\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269" Oct 30 21:53:06.932152 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:06.931840 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jrtr\" (UniqueName: \"kubernetes.io/projected/01205782-af7b-4eda-894d-3850de540135-kube-api-access-6jrtr\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269\" (UID: \"01205782-af7b-4eda-894d-3850de540135\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269" Oct 30 21:53:07.020936 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:07.020897 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:07.033060 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:07.033019 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01205782-af7b-4eda-894d-3850de540135-workload-socket\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269\" (UID: \"01205782-af7b-4eda-894d-3850de540135\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269" Oct 30 21:53:07.033213 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:07.033093 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01205782-af7b-4eda-894d-3850de540135-credential-socket\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269\" (UID: \"01205782-af7b-4eda-894d-3850de540135\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269" Oct 30 21:53:07.033213 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:07.033139 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01205782-af7b-4eda-894d-3850de540135-istio-podinfo\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269\" (UID: \"01205782-af7b-4eda-894d-3850de540135\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269" Oct 30 21:53:07.033213 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:07.033173 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01205782-af7b-4eda-894d-3850de540135-istio-data\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269\" (UID: \"01205782-af7b-4eda-894d-3850de540135\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269" Oct 30 21:53:07.033213 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:07.033203 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01205782-af7b-4eda-894d-3850de540135-istiod-ca-cert\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269\" (UID: \"01205782-af7b-4eda-894d-3850de540135\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269" Oct 30 21:53:07.033373 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:07.033243 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01205782-af7b-4eda-894d-3850de540135-istio-token\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269\" (UID: \"01205782-af7b-4eda-894d-3850de540135\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269" Oct 30 21:53:07.033373 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:07.033301 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01205782-af7b-4eda-894d-3850de540135-istio-envoy\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269\" (UID: \"01205782-af7b-4eda-894d-3850de540135\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269" Oct 30 21:53:07.033373 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:07.033356 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01205782-af7b-4eda-894d-3850de540135-workload-certs\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269\" (UID: \"01205782-af7b-4eda-894d-3850de540135\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269" Oct 30 21:53:07.033502 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:07.033460 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jrtr\" (UniqueName: \"kubernetes.io/projected/01205782-af7b-4eda-894d-3850de540135-kube-api-access-6jrtr\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269\" (UID: \"01205782-af7b-4eda-894d-3850de540135\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269" Oct 30 21:53:07.033975 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:07.033938 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01205782-af7b-4eda-894d-3850de540135-istio-data\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269\" (UID: \"01205782-af7b-4eda-894d-3850de540135\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269" Oct 30 21:53:07.034123 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:07.033954 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01205782-af7b-4eda-894d-3850de540135-credential-socket\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269\" (UID: \"01205782-af7b-4eda-894d-3850de540135\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269" Oct 30 21:53:07.034123 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:07.034075 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01205782-af7b-4eda-894d-3850de540135-workload-socket\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269\" (UID: \"01205782-af7b-4eda-894d-3850de540135\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269" Oct 30 21:53:07.034343 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:07.034322 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01205782-af7b-4eda-894d-3850de540135-workload-certs\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269\" (UID: \"01205782-af7b-4eda-894d-3850de540135\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269" Oct 30 21:53:07.035091 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:07.035063 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01205782-af7b-4eda-894d-3850de540135-istiod-ca-cert\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269\" (UID: \"01205782-af7b-4eda-894d-3850de540135\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269" Oct 30 21:53:07.036544 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:07.036514 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01205782-af7b-4eda-894d-3850de540135-istio-envoy\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269\" (UID: \"01205782-af7b-4eda-894d-3850de540135\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269" Oct 30 21:53:07.037075 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:07.037052 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01205782-af7b-4eda-894d-3850de540135-istio-podinfo\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269\" (UID: \"01205782-af7b-4eda-894d-3850de540135\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269" Oct 30 21:53:07.048982 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:07.048951 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jrtr\" (UniqueName: \"kubernetes.io/projected/01205782-af7b-4eda-894d-3850de540135-kube-api-access-6jrtr\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269\" (UID: \"01205782-af7b-4eda-894d-3850de540135\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269" Oct 30 21:53:07.049949 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:07.049927 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01205782-af7b-4eda-894d-3850de540135-istio-token\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269\" (UID: \"01205782-af7b-4eda-894d-3850de540135\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269" Oct 30 21:53:07.134727 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:07.134688 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-rvzw9"] Oct 30 21:53:07.140228 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:07.140200 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-rvzw9" Oct 30 21:53:07.152752 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:07.152725 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-ceb36a-openshift-default-dockercfg-vtb7n\"" Oct 30 21:53:07.169406 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:07.166701 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-rvzw9"] Oct 30 21:53:07.178814 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:07.178700 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269" Oct 30 21:53:07.241601 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:07.238942 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc1b52ec-a38c-409e-810c-7ae3998c2ebd-workload-certs\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-rvzw9\" (UID: \"cc1b52ec-a38c-409e-810c-7ae3998c2ebd\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-rvzw9" Oct 30 21:53:07.241601 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:07.239007 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc1b52ec-a38c-409e-810c-7ae3998c2ebd-workload-socket\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-rvzw9\" (UID: \"cc1b52ec-a38c-409e-810c-7ae3998c2ebd\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-rvzw9" Oct 30 21:53:07.241601 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:07.239068 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc1b52ec-a38c-409e-810c-7ae3998c2ebd-istio-data\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-rvzw9\" (UID: \"cc1b52ec-a38c-409e-810c-7ae3998c2ebd\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-rvzw9" Oct 30 21:53:07.241601 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:07.239097 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc1b52ec-a38c-409e-810c-7ae3998c2ebd-istiod-ca-cert\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-rvzw9\" (UID: \"cc1b52ec-a38c-409e-810c-7ae3998c2ebd\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-rvzw9" Oct 30 21:53:07.241601 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:07.239154 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vlpg\" (UniqueName: \"kubernetes.io/projected/cc1b52ec-a38c-409e-810c-7ae3998c2ebd-kube-api-access-5vlpg\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-rvzw9\" (UID: \"cc1b52ec-a38c-409e-810c-7ae3998c2ebd\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-rvzw9" Oct 30 21:53:07.241601 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:07.239198 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc1b52ec-a38c-409e-810c-7ae3998c2ebd-istio-podinfo\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-rvzw9\" (UID: \"cc1b52ec-a38c-409e-810c-7ae3998c2ebd\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-rvzw9" Oct 30 21:53:07.241601 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:07.239233 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc1b52ec-a38c-409e-810c-7ae3998c2ebd-istio-envoy\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-rvzw9\" (UID: \"cc1b52ec-a38c-409e-810c-7ae3998c2ebd\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-rvzw9" Oct 30 21:53:07.241601 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:07.239265 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc1b52ec-a38c-409e-810c-7ae3998c2ebd-credential-socket\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-rvzw9\" (UID: \"cc1b52ec-a38c-409e-810c-7ae3998c2ebd\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-rvzw9" Oct 30 21:53:07.241601 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:07.239292 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc1b52ec-a38c-409e-810c-7ae3998c2ebd-istio-token\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-rvzw9\" (UID: \"cc1b52ec-a38c-409e-810c-7ae3998c2ebd\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-rvzw9" Oct 30 21:53:07.340849 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:07.340811 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc1b52ec-a38c-409e-810c-7ae3998c2ebd-workload-certs\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-rvzw9\" (UID: \"cc1b52ec-a38c-409e-810c-7ae3998c2ebd\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-rvzw9" Oct 30 21:53:07.340849 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:07.340852 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc1b52ec-a38c-409e-810c-7ae3998c2ebd-workload-socket\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-rvzw9\" (UID: \"cc1b52ec-a38c-409e-810c-7ae3998c2ebd\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-rvzw9" Oct 30 21:53:07.341064 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:07.341041 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc1b52ec-a38c-409e-810c-7ae3998c2ebd-istio-data\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-rvzw9\" (UID: \"cc1b52ec-a38c-409e-810c-7ae3998c2ebd\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-rvzw9" Oct 30 21:53:07.341111 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:07.341088 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc1b52ec-a38c-409e-810c-7ae3998c2ebd-istiod-ca-cert\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-rvzw9\" (UID: \"cc1b52ec-a38c-409e-810c-7ae3998c2ebd\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-rvzw9" Oct 30 21:53:07.341178 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:07.341161 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vlpg\" (UniqueName: \"kubernetes.io/projected/cc1b52ec-a38c-409e-810c-7ae3998c2ebd-kube-api-access-5vlpg\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-rvzw9\" (UID: \"cc1b52ec-a38c-409e-810c-7ae3998c2ebd\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-rvzw9" Oct 30 21:53:07.341235 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:07.341209 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc1b52ec-a38c-409e-810c-7ae3998c2ebd-istio-podinfo\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-rvzw9\" (UID: \"cc1b52ec-a38c-409e-810c-7ae3998c2ebd\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-rvzw9" Oct 30 21:53:07.341285 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:07.341238 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc1b52ec-a38c-409e-810c-7ae3998c2ebd-workload-certs\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-rvzw9\" (UID: \"cc1b52ec-a38c-409e-810c-7ae3998c2ebd\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-rvzw9" Oct 30 21:53:07.341285 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:07.341249 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc1b52ec-a38c-409e-810c-7ae3998c2ebd-istio-envoy\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-rvzw9\" (UID: \"cc1b52ec-a38c-409e-810c-7ae3998c2ebd\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-rvzw9" Oct 30 21:53:07.341438 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:07.341287 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc1b52ec-a38c-409e-810c-7ae3998c2ebd-workload-socket\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-rvzw9\" (UID: \"cc1b52ec-a38c-409e-810c-7ae3998c2ebd\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-rvzw9" Oct 30 21:53:07.341438 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:07.341301 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc1b52ec-a38c-409e-810c-7ae3998c2ebd-credential-socket\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-rvzw9\" (UID: \"cc1b52ec-a38c-409e-810c-7ae3998c2ebd\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-rvzw9" Oct 30 21:53:07.341438 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:07.341320 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc1b52ec-a38c-409e-810c-7ae3998c2ebd-istio-data\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-rvzw9\" (UID: \"cc1b52ec-a38c-409e-810c-7ae3998c2ebd\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-rvzw9" Oct 30 21:53:07.341438 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:07.341344 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc1b52ec-a38c-409e-810c-7ae3998c2ebd-istio-token\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-rvzw9\" (UID: \"cc1b52ec-a38c-409e-810c-7ae3998c2ebd\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-rvzw9" Oct 30 21:53:07.341656 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:07.341635 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc1b52ec-a38c-409e-810c-7ae3998c2ebd-credential-socket\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-rvzw9\" (UID: \"cc1b52ec-a38c-409e-810c-7ae3998c2ebd\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-rvzw9" Oct 30 21:53:07.341878 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:07.341857 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc1b52ec-a38c-409e-810c-7ae3998c2ebd-istiod-ca-cert\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-rvzw9\" (UID: \"cc1b52ec-a38c-409e-810c-7ae3998c2ebd\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-rvzw9" Oct 30 21:53:07.343857 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:07.343836 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc1b52ec-a38c-409e-810c-7ae3998c2ebd-istio-envoy\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-rvzw9\" (UID: \"cc1b52ec-a38c-409e-810c-7ae3998c2ebd\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-rvzw9" Oct 30 21:53:07.343950 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:07.343934 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc1b52ec-a38c-409e-810c-7ae3998c2ebd-istio-podinfo\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-rvzw9\" (UID: \"cc1b52ec-a38c-409e-810c-7ae3998c2ebd\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-rvzw9" Oct 30 21:53:07.354455 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:07.354372 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vlpg\" (UniqueName: \"kubernetes.io/projected/cc1b52ec-a38c-409e-810c-7ae3998c2ebd-kube-api-access-5vlpg\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-rvzw9\" (UID: \"cc1b52ec-a38c-409e-810c-7ae3998c2ebd\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-rvzw9" Oct 30 21:53:07.354455 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:07.354419 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc1b52ec-a38c-409e-810c-7ae3998c2ebd-istio-token\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-rvzw9\" (UID: \"cc1b52ec-a38c-409e-810c-7ae3998c2ebd\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-rvzw9" Oct 30 21:53:07.463463 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:07.458554 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-rvzw9" Oct 30 21:53:07.496135 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:07.496104 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:07.520423 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:07.513115 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269"] Oct 30 21:53:07.532414 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:07.531804 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:53:07.532414 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:07.531894 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:53:07.532414 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:07.531941 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:53:07.770075 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:07.769968 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269" event={"ID":"01205782-af7b-4eda-894d-3850de540135","Type":"ContainerStarted","Data":"d96ef1c5779215cd09940284bd4b102d5ee5fcdd28f6c870dd4fcdca37681940"} Oct 30 21:53:07.770075 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:07.770034 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269" event={"ID":"01205782-af7b-4eda-894d-3850de540135","Type":"ContainerStarted","Data":"312cbe52ade4cdb38ea8d0398a1d742d4495fd2508c18d03d8c67983b4faecc4"} Oct 30 21:53:07.772241 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:07.772214 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-rvzw9"] Oct 30 21:53:07.781156 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:53:07.781126 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc1b52ec_a38c_409e_810c_7ae3998c2ebd.slice/crio-f947ff9093e901903eb19ede176ad88e4b67b1815b6569d8bf006986692ebec8 WatchSource:0}: Error finding container f947ff9093e901903eb19ede176ad88e4b67b1815b6569d8bf006986692ebec8: Status 404 returned error can't find the container with id f947ff9093e901903eb19ede176ad88e4b67b1815b6569d8bf006986692ebec8 Oct 30 21:53:07.784747 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:07.784718 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:53:07.784848 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:07.784778 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:53:07.784848 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:07.784804 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:53:07.799321 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:07.799266 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269" podStartSLOduration=1.799248532 podStartE2EDuration="1.799248532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:07.79710789 +0000 UTC m=+753.968073069" watchObservedRunningTime="2026-10-30 21:53:07.799248532 +0000 UTC m=+753.970213714" Oct 30 21:53:08.180522 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:08.180435 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269" Oct 30 21:53:08.181035 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:08.181001 6720 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 21:53:08.181103 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:08.181049 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269" podUID="01205782-af7b-4eda-894d-3850de540135" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 21:53:08.642297 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:08.642179 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:08.800503 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:08.800460 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-rvzw9" event={"ID":"cc1b52ec-a38c-409e-810c-7ae3998c2ebd","Type":"ContainerStarted","Data":"4c32c1e5c36e5394efc19774aceeb4b04f6086c5347f34532bbe4bd3d99edafe"} Oct 30 21:53:08.800717 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:08.800521 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-rvzw9" event={"ID":"cc1b52ec-a38c-409e-810c-7ae3998c2ebd","Type":"ContainerStarted","Data":"f947ff9093e901903eb19ede176ad88e4b67b1815b6569d8bf006986692ebec8"} Oct 30 21:53:08.827974 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:08.827923 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-rvzw9" podStartSLOduration=1.827909861 podStartE2EDuration="1.827909861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:08.826153067 +0000 UTC m=+754.997118250" watchObservedRunningTime="2026-10-30 21:53:08.827909861 +0000 UTC m=+754.998875043" Oct 30 21:53:09.179807 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:09.179768 6720 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 21:53:09.180107 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:09.180067 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269" podUID="01205782-af7b-4eda-894d-3850de540135" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 21:53:09.229766 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:09.229734 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-7t2lk"] Oct 30 21:53:09.233837 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:09.233816 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-7t2lk" Oct 30 21:53:09.238542 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:09.238521 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-939a16-openshift-default-dockercfg-6sr4l\"" Oct 30 21:53:09.265407 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:09.261853 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-7t2lk"] Oct 30 21:53:09.406450 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:09.403063 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0902dba2-ba96-4382-86b8-f332dd81a061-istio-podinfo\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-7t2lk\" (UID: \"0902dba2-ba96-4382-86b8-f332dd81a061\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-7t2lk" Oct 30 21:53:09.406450 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:09.403129 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rz9t\" (UniqueName: \"kubernetes.io/projected/0902dba2-ba96-4382-86b8-f332dd81a061-kube-api-access-5rz9t\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-7t2lk\" (UID: \"0902dba2-ba96-4382-86b8-f332dd81a061\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-7t2lk" Oct 30 21:53:09.406450 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:09.403214 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0902dba2-ba96-4382-86b8-f332dd81a061-istio-envoy\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-7t2lk\" (UID: \"0902dba2-ba96-4382-86b8-f332dd81a061\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-7t2lk" Oct 30 21:53:09.406450 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:09.403278 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0902dba2-ba96-4382-86b8-f332dd81a061-workload-socket\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-7t2lk\" (UID: \"0902dba2-ba96-4382-86b8-f332dd81a061\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-7t2lk" Oct 30 21:53:09.406450 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:09.403355 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0902dba2-ba96-4382-86b8-f332dd81a061-credential-socket\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-7t2lk\" (UID: \"0902dba2-ba96-4382-86b8-f332dd81a061\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-7t2lk" Oct 30 21:53:09.406450 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:09.403434 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0902dba2-ba96-4382-86b8-f332dd81a061-istio-token\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-7t2lk\" (UID: \"0902dba2-ba96-4382-86b8-f332dd81a061\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-7t2lk" Oct 30 21:53:09.406450 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:09.403460 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0902dba2-ba96-4382-86b8-f332dd81a061-istiod-ca-cert\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-7t2lk\" (UID: \"0902dba2-ba96-4382-86b8-f332dd81a061\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-7t2lk" Oct 30 21:53:09.406450 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:09.403533 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0902dba2-ba96-4382-86b8-f332dd81a061-istio-data\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-7t2lk\" (UID: \"0902dba2-ba96-4382-86b8-f332dd81a061\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-7t2lk" Oct 30 21:53:09.406450 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:09.403582 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0902dba2-ba96-4382-86b8-f332dd81a061-workload-certs\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-7t2lk\" (UID: \"0902dba2-ba96-4382-86b8-f332dd81a061\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-7t2lk" Oct 30 21:53:09.468464 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:09.467443 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-rvzw9" Oct 30 21:53:09.491408 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:09.471485 6720 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-rvzw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 21:53:09.491408 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:09.471538 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-rvzw9" podUID="cc1b52ec-a38c-409e-810c-7ae3998c2ebd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 21:53:09.506374 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:09.505255 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0902dba2-ba96-4382-86b8-f332dd81a061-istio-data\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-7t2lk\" (UID: \"0902dba2-ba96-4382-86b8-f332dd81a061\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-7t2lk" Oct 30 21:53:09.506374 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:09.505324 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0902dba2-ba96-4382-86b8-f332dd81a061-workload-certs\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-7t2lk\" (UID: \"0902dba2-ba96-4382-86b8-f332dd81a061\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-7t2lk" Oct 30 21:53:09.506374 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:09.505427 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0902dba2-ba96-4382-86b8-f332dd81a061-istio-podinfo\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-7t2lk\" (UID: \"0902dba2-ba96-4382-86b8-f332dd81a061\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-7t2lk" Oct 30 21:53:09.506374 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:09.505472 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rz9t\" (UniqueName: \"kubernetes.io/projected/0902dba2-ba96-4382-86b8-f332dd81a061-kube-api-access-5rz9t\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-7t2lk\" (UID: \"0902dba2-ba96-4382-86b8-f332dd81a061\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-7t2lk" Oct 30 21:53:09.506374 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:09.505546 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0902dba2-ba96-4382-86b8-f332dd81a061-istio-envoy\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-7t2lk\" (UID: \"0902dba2-ba96-4382-86b8-f332dd81a061\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-7t2lk" Oct 30 21:53:09.506374 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:09.505572 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0902dba2-ba96-4382-86b8-f332dd81a061-workload-socket\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-7t2lk\" (UID: \"0902dba2-ba96-4382-86b8-f332dd81a061\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-7t2lk" Oct 30 21:53:09.506374 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:09.505612 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0902dba2-ba96-4382-86b8-f332dd81a061-credential-socket\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-7t2lk\" (UID: \"0902dba2-ba96-4382-86b8-f332dd81a061\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-7t2lk" Oct 30 21:53:09.506374 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:09.505628 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0902dba2-ba96-4382-86b8-f332dd81a061-istio-token\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-7t2lk\" (UID: \"0902dba2-ba96-4382-86b8-f332dd81a061\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-7t2lk" Oct 30 21:53:09.506374 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:09.505647 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0902dba2-ba96-4382-86b8-f332dd81a061-istiod-ca-cert\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-7t2lk\" (UID: \"0902dba2-ba96-4382-86b8-f332dd81a061\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-7t2lk" Oct 30 21:53:09.506374 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:09.506240 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0902dba2-ba96-4382-86b8-f332dd81a061-istiod-ca-cert\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-7t2lk\" (UID: \"0902dba2-ba96-4382-86b8-f332dd81a061\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-7t2lk" Oct 30 21:53:09.507544 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:09.506776 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0902dba2-ba96-4382-86b8-f332dd81a061-credential-socket\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-7t2lk\" (UID: \"0902dba2-ba96-4382-86b8-f332dd81a061\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-7t2lk" Oct 30 21:53:09.507544 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:09.507202 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0902dba2-ba96-4382-86b8-f332dd81a061-workload-certs\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-7t2lk\" (UID: \"0902dba2-ba96-4382-86b8-f332dd81a061\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-7t2lk" Oct 30 21:53:09.507544 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:09.507317 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0902dba2-ba96-4382-86b8-f332dd81a061-istio-data\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-7t2lk\" (UID: \"0902dba2-ba96-4382-86b8-f332dd81a061\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-7t2lk" Oct 30 21:53:09.508055 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:09.507877 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0902dba2-ba96-4382-86b8-f332dd81a061-workload-socket\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-7t2lk\" (UID: \"0902dba2-ba96-4382-86b8-f332dd81a061\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-7t2lk" Oct 30 21:53:09.509926 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:09.509881 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0902dba2-ba96-4382-86b8-f332dd81a061-istio-envoy\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-7t2lk\" (UID: \"0902dba2-ba96-4382-86b8-f332dd81a061\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-7t2lk" Oct 30 21:53:09.510410 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:09.510376 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0902dba2-ba96-4382-86b8-f332dd81a061-istio-podinfo\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-7t2lk\" (UID: \"0902dba2-ba96-4382-86b8-f332dd81a061\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-7t2lk" Oct 30 21:53:09.522296 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:09.522272 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0902dba2-ba96-4382-86b8-f332dd81a061-istio-token\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-7t2lk\" (UID: \"0902dba2-ba96-4382-86b8-f332dd81a061\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-7t2lk" Oct 30 21:53:09.522800 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:09.522782 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rz9t\" (UniqueName: \"kubernetes.io/projected/0902dba2-ba96-4382-86b8-f332dd81a061-kube-api-access-5rz9t\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-7t2lk\" (UID: \"0902dba2-ba96-4382-86b8-f332dd81a061\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-7t2lk" Oct 30 21:53:09.614984 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:09.614953 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-7t2lk" Oct 30 21:53:09.615166 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:09.615148 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:09.902617 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:09.902570 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-ceb36a/e2e-conflict-a-dfac3fd4-kserve-677cc88846-vf5kf" podUID="ab5d437e-6b94-4c23-b45e-a93f8a840ffe" containerName="main" containerID="cri-o://ef17724ea2f40cb6a5d394ab0b3c0195a34835bc73e2242fb6573518fe9bb00a" gracePeriod=45 Oct 30 21:53:09.976958 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:09.976878 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-7t2lk"] Oct 30 21:53:10.012594 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:10.012555 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:53:10.012791 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:10.012770 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:53:10.012953 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:10.012933 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:53:10.227317 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:10.226131 6720 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 21:53:10.227317 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:10.226188 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269" podUID="01205782-af7b-4eda-894d-3850de540135" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 21:53:10.431115 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:10.431085 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ceb36a/e2e-conflict-a-dfac3fd4-kserve-677cc88846-vf5kf" Oct 30 21:53:10.435295 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:10.435270 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab5d437e-6b94-4c23-b45e-a93f8a840ffe-home\" (UniqueName: \"kubernetes.io/empty-dir/ab5d437e-6b94-4c23-b45e-a93f8a840ffe-home\") pod \"ab5d437e-6b94-4c23-b45e-a93f8a840ffe\" (UID: \"ab5d437e-6b94-4c23-b45e-a93f8a840ffe\") " Oct 30 21:53:10.435460 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:10.435333 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab5d437e-6b94-4c23-b45e-a93f8a840ffe-tls-certs\" (UniqueName: \"kubernetes.io/secret/ab5d437e-6b94-4c23-b45e-a93f8a840ffe-tls-certs\") pod \"ab5d437e-6b94-4c23-b45e-a93f8a840ffe\" (UID: \"ab5d437e-6b94-4c23-b45e-a93f8a840ffe\") " Oct 30 21:53:10.435460 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:10.435411 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab5d437e-6b94-4c23-b45e-a93f8a840ffe-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab5d437e-6b94-4c23-b45e-a93f8a840ffe-tmp-dir\") pod \"ab5d437e-6b94-4c23-b45e-a93f8a840ffe\" (UID: \"ab5d437e-6b94-4c23-b45e-a93f8a840ffe\") " Oct 30 21:53:10.435579 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:10.435461 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab5d437e-6b94-4c23-b45e-a93f8a840ffe-kube-api-access-t6rxq\" (UniqueName: \"kubernetes.io/projected/ab5d437e-6b94-4c23-b45e-a93f8a840ffe-kube-api-access-t6rxq\") pod \"ab5d437e-6b94-4c23-b45e-a93f8a840ffe\" (UID: \"ab5d437e-6b94-4c23-b45e-a93f8a840ffe\") " Oct 30 21:53:10.435579 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:10.435537 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab5d437e-6b94-4c23-b45e-a93f8a840ffe-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab5d437e-6b94-4c23-b45e-a93f8a840ffe-dshm\") pod \"ab5d437e-6b94-4c23-b45e-a93f8a840ffe\" (UID: \"ab5d437e-6b94-4c23-b45e-a93f8a840ffe\") " Oct 30 21:53:10.435679 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:10.435581 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab5d437e-6b94-4c23-b45e-a93f8a840ffe-home" pod "ab5d437e-6b94-4c23-b45e-a93f8a840ffe" (UID: "ab5d437e-6b94-4c23-b45e-a93f8a840ffe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:53:10.435679 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:10.435613 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab5d437e-6b94-4c23-b45e-a93f8a840ffe-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab5d437e-6b94-4c23-b45e-a93f8a840ffe-model-cache\") pod \"ab5d437e-6b94-4c23-b45e-a93f8a840ffe\" (UID: \"ab5d437e-6b94-4c23-b45e-a93f8a840ffe\") " Oct 30 21:53:10.436411 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:10.435780 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab5d437e-6b94-4c23-b45e-a93f8a840ffe-tmp-dir" pod "ab5d437e-6b94-4c23-b45e-a93f8a840ffe" (UID: "ab5d437e-6b94-4c23-b45e-a93f8a840ffe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:53:10.436411 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:10.435962 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab5d437e-6b94-4c23-b45e-a93f8a840ffe-model-cache" pod "ab5d437e-6b94-4c23-b45e-a93f8a840ffe" (UID: "ab5d437e-6b94-4c23-b45e-a93f8a840ffe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:53:10.436411 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:10.436093 6720 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab5d437e-6b94-4c23-b45e-a93f8a840ffe-home\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:53:10.436411 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:10.436113 6720 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab5d437e-6b94-4c23-b45e-a93f8a840ffe-tmp-dir\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:53:10.436411 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:10.436126 6720 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab5d437e-6b94-4c23-b45e-a93f8a840ffe-model-cache\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:53:10.438477 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:10.438454 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab5d437e-6b94-4c23-b45e-a93f8a840ffe-dshm" pod "ab5d437e-6b94-4c23-b45e-a93f8a840ffe" (UID: "ab5d437e-6b94-4c23-b45e-a93f8a840ffe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:53:10.438607 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:10.438583 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab5d437e-6b94-4c23-b45e-a93f8a840ffe-kube-api-access-t6rxq" pod "ab5d437e-6b94-4c23-b45e-a93f8a840ffe" (UID: "ab5d437e-6b94-4c23-b45e-a93f8a840ffe"). InnerVolumeSpecName "kube-api-access-t6rxq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:53:10.438676 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:10.438652 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab5d437e-6b94-4c23-b45e-a93f8a840ffe-tls-certs" pod "ab5d437e-6b94-4c23-b45e-a93f8a840ffe" (UID: "ab5d437e-6b94-4c23-b45e-a93f8a840ffe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:53:10.440586 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:53:10.440553 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 21:53:10.508004 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:10.507497 6720 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-rvzw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 21:53:10.508004 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:10.507561 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-rvzw9" podUID="cc1b52ec-a38c-409e-810c-7ae3998c2ebd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 21:53:10.536983 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:10.536947 6720 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab5d437e-6b94-4c23-b45e-a93f8a840ffe-tls-certs\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:53:10.536983 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:10.536978 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6rxq\" (UniqueName: \"kubernetes.io/projected/ab5d437e-6b94-4c23-b45e-a93f8a840ffe-kube-api-access-t6rxq\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:53:10.537201 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:10.537007 6720 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab5d437e-6b94-4c23-b45e-a93f8a840ffe-dshm\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:53:10.599407 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:10.598981 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:10.820139 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:10.820053 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ceb36a/e2e-conflict-a-dfac3fd4-kserve-677cc88846-vf5kf" Oct 30 21:53:10.820139 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:10.820088 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ceb36a/e2e-conflict-a-dfac3fd4-kserve-677cc88846-vf5kf" event={"ID":"ab5d437e-6b94-4c23-b45e-a93f8a840ffe","Type":"ContainerDied","Data":"ef17724ea2f40cb6a5d394ab0b3c0195a34835bc73e2242fb6573518fe9bb00a"} Oct 30 21:53:10.820337 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:10.820147 6720 scope.go:122] "RemoveContainer" containerID="ef17724ea2f40cb6a5d394ab0b3c0195a34835bc73e2242fb6573518fe9bb00a" Oct 30 21:53:10.825187 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:10.819968 6720 generic.go:356] "Generic (PLEG): container finished" podID="ab5d437e-6b94-4c23-b45e-a93f8a840ffe" containerID="ef17724ea2f40cb6a5d394ab0b3c0195a34835bc73e2242fb6573518fe9bb00a" exitCode=0 Oct 30 21:53:10.825436 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:10.825410 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ceb36a/e2e-conflict-a-dfac3fd4-kserve-677cc88846-vf5kf" event={"ID":"ab5d437e-6b94-4c23-b45e-a93f8a840ffe","Type":"ContainerDied","Data":"de06cf6404b78614377513ee259c55797246c1ba0086009683d9798462c00830"} Oct 30 21:53:10.828111 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:10.828082 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-7t2lk" event={"ID":"0902dba2-ba96-4382-86b8-f332dd81a061","Type":"ContainerStarted","Data":"9db27ae96858ba075fc3e0f44706e73e277d88de5a1acbb018f1b609dda51c7b"} Oct 30 21:53:10.828207 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:10.828124 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-7t2lk" event={"ID":"0902dba2-ba96-4382-86b8-f332dd81a061","Type":"ContainerStarted","Data":"17811af6a3b46b669481c6862e1c3add3700ee146fbbb402717db3ae5972c2b1"} Oct 30 21:53:10.831640 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:10.830425 6720 scope.go:122] "RemoveContainer" containerID="ef17724ea2f40cb6a5d394ab0b3c0195a34835bc73e2242fb6573518fe9bb00a" Oct 30 21:53:10.831640 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:53:10.830687 6720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef17724ea2f40cb6a5d394ab0b3c0195a34835bc73e2242fb6573518fe9bb00a\": container with ID starting with ef17724ea2f40cb6a5d394ab0b3c0195a34835bc73e2242fb6573518fe9bb00a not found: ID does not exist" containerID="ef17724ea2f40cb6a5d394ab0b3c0195a34835bc73e2242fb6573518fe9bb00a" Oct 30 21:53:10.831640 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:10.830715 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef17724ea2f40cb6a5d394ab0b3c0195a34835bc73e2242fb6573518fe9bb00a"} err="failed to get container status \"ef17724ea2f40cb6a5d394ab0b3c0195a34835bc73e2242fb6573518fe9bb00a\": rpc error: code = NotFound desc = could not find container \"ef17724ea2f40cb6a5d394ab0b3c0195a34835bc73e2242fb6573518fe9bb00a\": container with ID starting with ef17724ea2f40cb6a5d394ab0b3c0195a34835bc73e2242fb6573518fe9bb00a not found: ID does not exist" Oct 30 21:53:10.857117 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:10.857054 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-7t2lk" podStartSLOduration=1.857032975 podStartE2EDuration="1.857032975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:10.854981972 +0000 UTC m=+757.025947155" watchObservedRunningTime="2026-10-30 21:53:10.857032975 +0000 UTC m=+757.027998157" Oct 30 21:53:10.891320 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:10.890149 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-ceb36a/e2e-conflict-a-dfac3fd4-kserve-677cc88846-vf5kf"] Oct 30 21:53:10.891320 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:10.890204 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-ceb36a/e2e-conflict-a-dfac3fd4-kserve-677cc88846-vf5kf"] Oct 30 21:53:11.183412 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:11.179312 6720 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 21:53:11.183412 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:11.179372 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269" podUID="01205782-af7b-4eda-894d-3850de540135" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 21:53:11.403410 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:11.402849 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/e2e-conflict-a-dfac3fd4-kserve-677cc88846-vf5kf" podUID="ab5d437e-6b94-4c23-b45e-a93f8a840ffe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 21:53:11.465478 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:11.462800 6720 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-rvzw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 21:53:11.465478 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:11.462861 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-rvzw9" podUID="cc1b52ec-a38c-409e-810c-7ae3998c2ebd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 21:53:11.622410 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:11.616992 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-7t2lk" Oct 30 21:53:11.622410 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:11.617109 6720 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-7t2lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 21:53:11.622410 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:11.617152 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-7t2lk" podUID="0902dba2-ba96-4382-86b8-f332dd81a061" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 21:53:11.769474 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:11.768204 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:12.001933 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:12.001881 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab5d437e-6b94-4c23-b45e-a93f8a840ffe" path="/var/lib/kubelet/pods/ab5d437e-6b94-4c23-b45e-a93f8a840ffe/volumes" Oct 30 21:53:12.184801 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:12.184764 6720 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 21:53:12.185102 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:12.185069 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269" podUID="01205782-af7b-4eda-894d-3850de540135" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 21:53:12.460462 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:12.459779 6720 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-rvzw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 21:53:12.460462 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:12.459843 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-rvzw9" podUID="cc1b52ec-a38c-409e-810c-7ae3998c2ebd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 21:53:12.616779 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:12.616740 6720 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-7t2lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 21:53:12.617352 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:12.617315 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-7t2lk" podUID="0902dba2-ba96-4382-86b8-f332dd81a061" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 21:53:13.179806 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:13.179769 6720 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 21:53:13.180088 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:13.180061 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269" podUID="01205782-af7b-4eda-894d-3850de540135" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 21:53:13.465071 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:13.464976 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:13.465858 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:13.465773 6720 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-rvzw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 21:53:13.466029 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:13.466006 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-rvzw9" podUID="cc1b52ec-a38c-409e-810c-7ae3998c2ebd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 21:53:13.620882 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:13.620842 6720 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-7t2lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 21:53:13.621468 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:13.621432 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-7t2lk" podUID="0902dba2-ba96-4382-86b8-f332dd81a061" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 21:53:14.180987 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:14.180949 6720 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 21:53:14.181360 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:14.181308 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269" podUID="01205782-af7b-4eda-894d-3850de540135" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 21:53:14.477804 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:14.477675 6720 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-rvzw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 21:53:14.478098 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:14.478049 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-rvzw9" podUID="cc1b52ec-a38c-409e-810c-7ae3998c2ebd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 21:53:14.532761 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:14.532728 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:14.616120 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:14.616078 6720 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-7t2lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 21:53:14.616330 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:14.616144 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-7t2lk" podUID="0902dba2-ba96-4382-86b8-f332dd81a061" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 21:53:15.328295 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:15.328265 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269" Oct 30 21:53:15.329391 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:15.329364 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269" Oct 30 21:53:15.330303 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:15.330286 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269" Oct 30 21:53:15.434081 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:15.434051 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:15.473702 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:15.473664 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-rvzw9" Oct 30 21:53:15.474891 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:15.474874 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-rvzw9" Oct 30 21:53:15.475557 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:15.475537 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-rvzw9" Oct 30 21:53:15.615482 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:15.615446 6720 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-7t2lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 21:53:15.615629 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:15.615508 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-7t2lk" podUID="0902dba2-ba96-4382-86b8-f332dd81a061" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 21:53:16.353872 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:16.353788 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:16.615346 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:16.615264 6720 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-7t2lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 21:53:16.615346 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:16.615332 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-7t2lk" podUID="0902dba2-ba96-4382-86b8-f332dd81a061" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 21:53:17.194135 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:17.194103 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:17.895077 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:17.895052 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-7t2lk" Oct 30 21:53:17.896697 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:17.896022 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-7t2lk" Oct 30 21:53:17.897059 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:17.897039 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-7t2lk" Oct 30 21:53:18.254562 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:18.254478 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:18.909559 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:18.909528 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:19.076527 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:19.076497 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:19.273561 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:19.273486 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:20.282188 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:20.282156 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:21.106169 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:21.106140 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:21.833068 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:21.833030 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:21.894745 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:21.894706 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9"] Oct 30 21:53:21.895225 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:21.895210 6720 cpu_manager.go:414] "containerMap: removing container" podUID="ab5d437e-6b94-4c23-b45e-a93f8a840ffe" containerName="main" Oct 30 21:53:21.895271 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:21.895228 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab5d437e-6b94-4c23-b45e-a93f8a840ffe" containerName="main" Oct 30 21:53:21.895344 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:21.895335 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab5d437e-6b94-4c23-b45e-a93f8a840ffe" containerName="main" Oct 30 21:53:21.898524 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:21.898470 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9" Oct 30 21:53:21.914668 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:21.914641 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9"] Oct 30 21:53:22.009466 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:22.009431 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13913563-7222-407c-98b1-5c1a17d10b82-istio-token\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9\" (UID: \"13913563-7222-407c-98b1-5c1a17d10b82\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9" Oct 30 21:53:22.009612 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:22.009539 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13913563-7222-407c-98b1-5c1a17d10b82-workload-certs\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9\" (UID: \"13913563-7222-407c-98b1-5c1a17d10b82\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9" Oct 30 21:53:22.009612 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:22.009581 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13913563-7222-407c-98b1-5c1a17d10b82-istiod-ca-cert\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9\" (UID: \"13913563-7222-407c-98b1-5c1a17d10b82\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9" Oct 30 21:53:22.009694 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:22.009667 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13913563-7222-407c-98b1-5c1a17d10b82-istio-envoy\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9\" (UID: \"13913563-7222-407c-98b1-5c1a17d10b82\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9" Oct 30 21:53:22.009731 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:22.009705 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13913563-7222-407c-98b1-5c1a17d10b82-credential-socket\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9\" (UID: \"13913563-7222-407c-98b1-5c1a17d10b82\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9" Oct 30 21:53:22.009790 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:22.009771 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13913563-7222-407c-98b1-5c1a17d10b82-workload-socket\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9\" (UID: \"13913563-7222-407c-98b1-5c1a17d10b82\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9" Oct 30 21:53:22.009828 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:22.009809 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13913563-7222-407c-98b1-5c1a17d10b82-istio-data\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9\" (UID: \"13913563-7222-407c-98b1-5c1a17d10b82\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9" Oct 30 21:53:22.009914 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:22.009902 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13913563-7222-407c-98b1-5c1a17d10b82-istio-podinfo\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9\" (UID: \"13913563-7222-407c-98b1-5c1a17d10b82\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9" Oct 30 21:53:22.009965 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:22.009955 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg84w\" (UniqueName: \"kubernetes.io/projected/13913563-7222-407c-98b1-5c1a17d10b82-kube-api-access-sg84w\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9\" (UID: \"13913563-7222-407c-98b1-5c1a17d10b82\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9" Oct 30 21:53:22.110754 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:22.110713 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg84w\" (UniqueName: \"kubernetes.io/projected/13913563-7222-407c-98b1-5c1a17d10b82-kube-api-access-sg84w\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9\" (UID: \"13913563-7222-407c-98b1-5c1a17d10b82\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9" Oct 30 21:53:22.110983 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:22.110773 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13913563-7222-407c-98b1-5c1a17d10b82-istio-token\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9\" (UID: \"13913563-7222-407c-98b1-5c1a17d10b82\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9" Oct 30 21:53:22.111139 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:22.110985 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13913563-7222-407c-98b1-5c1a17d10b82-workload-certs\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9\" (UID: \"13913563-7222-407c-98b1-5c1a17d10b82\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9" Oct 30 21:53:22.111139 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:22.111049 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13913563-7222-407c-98b1-5c1a17d10b82-istiod-ca-cert\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9\" (UID: \"13913563-7222-407c-98b1-5c1a17d10b82\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9" Oct 30 21:53:22.111139 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:22.111129 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13913563-7222-407c-98b1-5c1a17d10b82-istio-envoy\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9\" (UID: \"13913563-7222-407c-98b1-5c1a17d10b82\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9" Oct 30 21:53:22.112276 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:22.111186 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13913563-7222-407c-98b1-5c1a17d10b82-credential-socket\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9\" (UID: \"13913563-7222-407c-98b1-5c1a17d10b82\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9" Oct 30 21:53:22.112276 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:22.111249 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13913563-7222-407c-98b1-5c1a17d10b82-workload-socket\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9\" (UID: \"13913563-7222-407c-98b1-5c1a17d10b82\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9" Oct 30 21:53:22.112276 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:22.111287 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13913563-7222-407c-98b1-5c1a17d10b82-istio-data\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9\" (UID: \"13913563-7222-407c-98b1-5c1a17d10b82\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9" Oct 30 21:53:22.112276 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:22.111749 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13913563-7222-407c-98b1-5c1a17d10b82-istio-data\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9\" (UID: \"13913563-7222-407c-98b1-5c1a17d10b82\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9" Oct 30 21:53:22.112276 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:22.111811 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13913563-7222-407c-98b1-5c1a17d10b82-istio-podinfo\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9\" (UID: \"13913563-7222-407c-98b1-5c1a17d10b82\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9" Oct 30 21:53:22.112276 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:22.111869 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13913563-7222-407c-98b1-5c1a17d10b82-workload-certs\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9\" (UID: \"13913563-7222-407c-98b1-5c1a17d10b82\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9" Oct 30 21:53:22.112276 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:22.111932 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13913563-7222-407c-98b1-5c1a17d10b82-credential-socket\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9\" (UID: \"13913563-7222-407c-98b1-5c1a17d10b82\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9" Oct 30 21:53:22.112276 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:22.112200 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13913563-7222-407c-98b1-5c1a17d10b82-workload-socket\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9\" (UID: \"13913563-7222-407c-98b1-5c1a17d10b82\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9" Oct 30 21:53:22.112563 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:22.112441 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13913563-7222-407c-98b1-5c1a17d10b82-istiod-ca-cert\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9\" (UID: \"13913563-7222-407c-98b1-5c1a17d10b82\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9" Oct 30 21:53:22.114117 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:22.114096 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13913563-7222-407c-98b1-5c1a17d10b82-istio-envoy\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9\" (UID: \"13913563-7222-407c-98b1-5c1a17d10b82\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9" Oct 30 21:53:22.114444 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:22.114426 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13913563-7222-407c-98b1-5c1a17d10b82-istio-podinfo\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9\" (UID: \"13913563-7222-407c-98b1-5c1a17d10b82\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9" Oct 30 21:53:22.134671 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:22.134644 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13913563-7222-407c-98b1-5c1a17d10b82-istio-token\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9\" (UID: \"13913563-7222-407c-98b1-5c1a17d10b82\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9" Oct 30 21:53:22.134949 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:22.134925 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg84w\" (UniqueName: \"kubernetes.io/projected/13913563-7222-407c-98b1-5c1a17d10b82-kube-api-access-sg84w\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9\" (UID: \"13913563-7222-407c-98b1-5c1a17d10b82\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9" Oct 30 21:53:22.218721 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:22.218692 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9" Oct 30 21:53:22.526004 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:22.525818 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9"] Oct 30 21:53:22.532618 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:22.532575 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:53:22.532760 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:22.532734 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:53:22.532857 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:22.532841 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:53:22.753038 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:22.753007 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:22.964217 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:22.964183 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9" event={"ID":"13913563-7222-407c-98b1-5c1a17d10b82","Type":"ContainerStarted","Data":"13d0f98a7a026a629eb0f6f0c23d0fbd558e7ca662aa0a669e51da5a049520a4"} Oct 30 21:53:22.964651 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:22.964239 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9" event={"ID":"13913563-7222-407c-98b1-5c1a17d10b82","Type":"ContainerStarted","Data":"38b4b9210ac7ec93935a3efa37cde37887e7dfb3384f4e6f86a6e61dc0ed35e0"} Oct 30 21:53:22.993323 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:22.993277 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9" podStartSLOduration=1.993262631 podStartE2EDuration="1.993262631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:22.990103316 +0000 UTC m=+769.161068499" watchObservedRunningTime="2026-10-30 21:53:22.993262631 +0000 UTC m=+769.164227812" Oct 30 21:53:23.219154 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:23.219068 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9" Oct 30 21:53:23.220845 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:23.220818 6720 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 21:53:23.220916 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:23.220875 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9" podUID="13913563-7222-407c-98b1-5c1a17d10b82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 21:53:23.483372 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:23.483271 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:24.031605 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:24.031574 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-ceb36a/distinct-2-w3-ceb36a-kserve-7c59695467-46gdc"] Oct 30 21:53:24.035659 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:24.035639 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-2-w3-ceb36a-kserve-7c59695467-46gdc" Oct 30 21:53:24.056771 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:24.056185 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-ceb36a\"/\"distinct-2-w3-ceb36a-kserve-self-signed-certs\"" Oct 30 21:53:24.074887 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:24.070862 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-ceb36a/distinct-2-w3-ceb36a-kserve-7c59695467-46gdc"] Oct 30 21:53:24.133886 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:24.133850 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1b6e92c-2952-4bcb-ae33-91e7271342fe-model-cache\") pod \"distinct-2-w3-ceb36a-kserve-7c59695467-46gdc\" (UID: \"f1b6e92c-2952-4bcb-ae33-91e7271342fe\") " pod="e2e-models-e2e-worker-w3-ceb36a/distinct-2-w3-ceb36a-kserve-7c59695467-46gdc" Oct 30 21:53:24.134110 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:24.133933 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqxrv\" (UniqueName: \"kubernetes.io/projected/f1b6e92c-2952-4bcb-ae33-91e7271342fe-kube-api-access-lqxrv\") pod \"distinct-2-w3-ceb36a-kserve-7c59695467-46gdc\" (UID: \"f1b6e92c-2952-4bcb-ae33-91e7271342fe\") " pod="e2e-models-e2e-worker-w3-ceb36a/distinct-2-w3-ceb36a-kserve-7c59695467-46gdc" Oct 30 21:53:24.134110 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:24.134031 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1b6e92c-2952-4bcb-ae33-91e7271342fe-dshm\") pod \"distinct-2-w3-ceb36a-kserve-7c59695467-46gdc\" (UID: \"f1b6e92c-2952-4bcb-ae33-91e7271342fe\") " pod="e2e-models-e2e-worker-w3-ceb36a/distinct-2-w3-ceb36a-kserve-7c59695467-46gdc" Oct 30 21:53:24.134272 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:24.134109 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1b6e92c-2952-4bcb-ae33-91e7271342fe-tls-certs\") pod \"distinct-2-w3-ceb36a-kserve-7c59695467-46gdc\" (UID: \"f1b6e92c-2952-4bcb-ae33-91e7271342fe\") " pod="e2e-models-e2e-worker-w3-ceb36a/distinct-2-w3-ceb36a-kserve-7c59695467-46gdc" Oct 30 21:53:24.134272 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:24.134168 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1b6e92c-2952-4bcb-ae33-91e7271342fe-home\") pod \"distinct-2-w3-ceb36a-kserve-7c59695467-46gdc\" (UID: \"f1b6e92c-2952-4bcb-ae33-91e7271342fe\") " pod="e2e-models-e2e-worker-w3-ceb36a/distinct-2-w3-ceb36a-kserve-7c59695467-46gdc" Oct 30 21:53:24.134272 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:24.134208 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1b6e92c-2952-4bcb-ae33-91e7271342fe-tmp-dir\") pod \"distinct-2-w3-ceb36a-kserve-7c59695467-46gdc\" (UID: \"f1b6e92c-2952-4bcb-ae33-91e7271342fe\") " pod="e2e-models-e2e-worker-w3-ceb36a/distinct-2-w3-ceb36a-kserve-7c59695467-46gdc" Oct 30 21:53:24.221740 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:24.221705 6720 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 21:53:24.221903 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:24.221758 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9" podUID="13913563-7222-407c-98b1-5c1a17d10b82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 21:53:24.235962 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:24.235919 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1b6e92c-2952-4bcb-ae33-91e7271342fe-dshm\") pod \"distinct-2-w3-ceb36a-kserve-7c59695467-46gdc\" (UID: \"f1b6e92c-2952-4bcb-ae33-91e7271342fe\") " pod="e2e-models-e2e-worker-w3-ceb36a/distinct-2-w3-ceb36a-kserve-7c59695467-46gdc" Oct 30 21:53:24.236154 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:24.236005 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1b6e92c-2952-4bcb-ae33-91e7271342fe-tls-certs\") pod \"distinct-2-w3-ceb36a-kserve-7c59695467-46gdc\" (UID: \"f1b6e92c-2952-4bcb-ae33-91e7271342fe\") " pod="e2e-models-e2e-worker-w3-ceb36a/distinct-2-w3-ceb36a-kserve-7c59695467-46gdc" Oct 30 21:53:24.236154 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:24.236030 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1b6e92c-2952-4bcb-ae33-91e7271342fe-home\") pod \"distinct-2-w3-ceb36a-kserve-7c59695467-46gdc\" (UID: \"f1b6e92c-2952-4bcb-ae33-91e7271342fe\") " pod="e2e-models-e2e-worker-w3-ceb36a/distinct-2-w3-ceb36a-kserve-7c59695467-46gdc" Oct 30 21:53:24.236154 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:24.236053 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1b6e92c-2952-4bcb-ae33-91e7271342fe-tmp-dir\") pod \"distinct-2-w3-ceb36a-kserve-7c59695467-46gdc\" (UID: \"f1b6e92c-2952-4bcb-ae33-91e7271342fe\") " pod="e2e-models-e2e-worker-w3-ceb36a/distinct-2-w3-ceb36a-kserve-7c59695467-46gdc" Oct 30 21:53:24.236154 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:24.236114 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1b6e92c-2952-4bcb-ae33-91e7271342fe-model-cache\") pod \"distinct-2-w3-ceb36a-kserve-7c59695467-46gdc\" (UID: \"f1b6e92c-2952-4bcb-ae33-91e7271342fe\") " pod="e2e-models-e2e-worker-w3-ceb36a/distinct-2-w3-ceb36a-kserve-7c59695467-46gdc" Oct 30 21:53:24.236376 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:24.236169 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqxrv\" (UniqueName: \"kubernetes.io/projected/f1b6e92c-2952-4bcb-ae33-91e7271342fe-kube-api-access-lqxrv\") pod \"distinct-2-w3-ceb36a-kserve-7c59695467-46gdc\" (UID: \"f1b6e92c-2952-4bcb-ae33-91e7271342fe\") " pod="e2e-models-e2e-worker-w3-ceb36a/distinct-2-w3-ceb36a-kserve-7c59695467-46gdc" Oct 30 21:53:24.236913 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:24.236864 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1b6e92c-2952-4bcb-ae33-91e7271342fe-home\") pod \"distinct-2-w3-ceb36a-kserve-7c59695467-46gdc\" (UID: \"f1b6e92c-2952-4bcb-ae33-91e7271342fe\") " pod="e2e-models-e2e-worker-w3-ceb36a/distinct-2-w3-ceb36a-kserve-7c59695467-46gdc" Oct 30 21:53:24.237036 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:24.236940 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1b6e92c-2952-4bcb-ae33-91e7271342fe-tmp-dir\") pod \"distinct-2-w3-ceb36a-kserve-7c59695467-46gdc\" (UID: \"f1b6e92c-2952-4bcb-ae33-91e7271342fe\") " pod="e2e-models-e2e-worker-w3-ceb36a/distinct-2-w3-ceb36a-kserve-7c59695467-46gdc" Oct 30 21:53:24.237230 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:24.237202 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1b6e92c-2952-4bcb-ae33-91e7271342fe-model-cache\") pod \"distinct-2-w3-ceb36a-kserve-7c59695467-46gdc\" (UID: \"f1b6e92c-2952-4bcb-ae33-91e7271342fe\") " pod="e2e-models-e2e-worker-w3-ceb36a/distinct-2-w3-ceb36a-kserve-7c59695467-46gdc" Oct 30 21:53:24.237911 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:24.237888 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:24.239558 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:24.239533 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1b6e92c-2952-4bcb-ae33-91e7271342fe-dshm\") pod \"distinct-2-w3-ceb36a-kserve-7c59695467-46gdc\" (UID: \"f1b6e92c-2952-4bcb-ae33-91e7271342fe\") " pod="e2e-models-e2e-worker-w3-ceb36a/distinct-2-w3-ceb36a-kserve-7c59695467-46gdc" Oct 30 21:53:24.239753 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:24.239706 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1b6e92c-2952-4bcb-ae33-91e7271342fe-tls-certs\") pod \"distinct-2-w3-ceb36a-kserve-7c59695467-46gdc\" (UID: \"f1b6e92c-2952-4bcb-ae33-91e7271342fe\") " pod="e2e-models-e2e-worker-w3-ceb36a/distinct-2-w3-ceb36a-kserve-7c59695467-46gdc" Oct 30 21:53:24.266182 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:24.266104 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqxrv\" (UniqueName: \"kubernetes.io/projected/f1b6e92c-2952-4bcb-ae33-91e7271342fe-kube-api-access-lqxrv\") pod \"distinct-2-w3-ceb36a-kserve-7c59695467-46gdc\" (UID: \"f1b6e92c-2952-4bcb-ae33-91e7271342fe\") " pod="e2e-models-e2e-worker-w3-ceb36a/distinct-2-w3-ceb36a-kserve-7c59695467-46gdc" Oct 30 21:53:24.332788 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:24.331844 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-cjmnk"] Oct 30 21:53:24.336763 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:24.336714 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-cjmnk" Oct 30 21:53:24.394434 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:24.383191 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-cjmnk"] Oct 30 21:53:24.394434 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:24.385102 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-2-w3-ceb36a-kserve-7c59695467-46gdc" Oct 30 21:53:24.438432 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:24.438400 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5359ba6b-7deb-4055-9069-1e59a9c4e392-istio-data\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-cjmnk\" (UID: \"5359ba6b-7deb-4055-9069-1e59a9c4e392\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-cjmnk" Oct 30 21:53:24.438566 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:24.438462 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5359ba6b-7deb-4055-9069-1e59a9c4e392-workload-socket\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-cjmnk\" (UID: \"5359ba6b-7deb-4055-9069-1e59a9c4e392\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-cjmnk" Oct 30 21:53:24.438566 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:24.438496 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5359ba6b-7deb-4055-9069-1e59a9c4e392-istio-podinfo\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-cjmnk\" (UID: \"5359ba6b-7deb-4055-9069-1e59a9c4e392\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-cjmnk" Oct 30 21:53:24.438566 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:24.438513 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98jht\" (UniqueName: \"kubernetes.io/projected/5359ba6b-7deb-4055-9069-1e59a9c4e392-kube-api-access-98jht\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-cjmnk\" (UID: \"5359ba6b-7deb-4055-9069-1e59a9c4e392\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-cjmnk" Oct 30 21:53:24.438566 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:24.438546 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5359ba6b-7deb-4055-9069-1e59a9c4e392-istio-envoy\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-cjmnk\" (UID: \"5359ba6b-7deb-4055-9069-1e59a9c4e392\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-cjmnk" Oct 30 21:53:24.438704 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:24.438577 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5359ba6b-7deb-4055-9069-1e59a9c4e392-credential-socket\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-cjmnk\" (UID: \"5359ba6b-7deb-4055-9069-1e59a9c4e392\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-cjmnk" Oct 30 21:53:24.438704 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:24.438625 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5359ba6b-7deb-4055-9069-1e59a9c4e392-istio-token\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-cjmnk\" (UID: \"5359ba6b-7deb-4055-9069-1e59a9c4e392\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-cjmnk" Oct 30 21:53:24.438704 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:24.438652 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5359ba6b-7deb-4055-9069-1e59a9c4e392-istiod-ca-cert\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-cjmnk\" (UID: \"5359ba6b-7deb-4055-9069-1e59a9c4e392\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-cjmnk" Oct 30 21:53:24.438704 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:24.438682 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5359ba6b-7deb-4055-9069-1e59a9c4e392-workload-certs\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-cjmnk\" (UID: \"5359ba6b-7deb-4055-9069-1e59a9c4e392\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-cjmnk" Oct 30 21:53:24.540410 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:24.540322 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5359ba6b-7deb-4055-9069-1e59a9c4e392-workload-certs\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-cjmnk\" (UID: \"5359ba6b-7deb-4055-9069-1e59a9c4e392\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-cjmnk" Oct 30 21:53:24.540410 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:24.540372 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5359ba6b-7deb-4055-9069-1e59a9c4e392-istio-data\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-cjmnk\" (UID: \"5359ba6b-7deb-4055-9069-1e59a9c4e392\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-cjmnk" Oct 30 21:53:24.540599 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:24.540456 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5359ba6b-7deb-4055-9069-1e59a9c4e392-workload-socket\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-cjmnk\" (UID: \"5359ba6b-7deb-4055-9069-1e59a9c4e392\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-cjmnk" Oct 30 21:53:24.540599 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:24.540502 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5359ba6b-7deb-4055-9069-1e59a9c4e392-istio-podinfo\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-cjmnk\" (UID: \"5359ba6b-7deb-4055-9069-1e59a9c4e392\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-cjmnk" Oct 30 21:53:24.540599 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:24.540531 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98jht\" (UniqueName: \"kubernetes.io/projected/5359ba6b-7deb-4055-9069-1e59a9c4e392-kube-api-access-98jht\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-cjmnk\" (UID: \"5359ba6b-7deb-4055-9069-1e59a9c4e392\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-cjmnk" Oct 30 21:53:24.540599 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:24.540594 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5359ba6b-7deb-4055-9069-1e59a9c4e392-istio-envoy\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-cjmnk\" (UID: \"5359ba6b-7deb-4055-9069-1e59a9c4e392\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-cjmnk" Oct 30 21:53:24.540726 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:24.540644 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5359ba6b-7deb-4055-9069-1e59a9c4e392-credential-socket\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-cjmnk\" (UID: \"5359ba6b-7deb-4055-9069-1e59a9c4e392\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-cjmnk" Oct 30 21:53:24.540796 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:24.540724 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5359ba6b-7deb-4055-9069-1e59a9c4e392-istio-token\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-cjmnk\" (UID: \"5359ba6b-7deb-4055-9069-1e59a9c4e392\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-cjmnk" Oct 30 21:53:24.540796 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:24.540766 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5359ba6b-7deb-4055-9069-1e59a9c4e392-istiod-ca-cert\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-cjmnk\" (UID: \"5359ba6b-7deb-4055-9069-1e59a9c4e392\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-cjmnk" Oct 30 21:53:24.541758 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:24.541617 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5359ba6b-7deb-4055-9069-1e59a9c4e392-istiod-ca-cert\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-cjmnk\" (UID: \"5359ba6b-7deb-4055-9069-1e59a9c4e392\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-cjmnk" Oct 30 21:53:24.542981 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:24.541906 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5359ba6b-7deb-4055-9069-1e59a9c4e392-workload-certs\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-cjmnk\" (UID: \"5359ba6b-7deb-4055-9069-1e59a9c4e392\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-cjmnk" Oct 30 21:53:24.542981 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:24.542125 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5359ba6b-7deb-4055-9069-1e59a9c4e392-istio-data\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-cjmnk\" (UID: \"5359ba6b-7deb-4055-9069-1e59a9c4e392\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-cjmnk" Oct 30 21:53:24.542981 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:24.542335 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5359ba6b-7deb-4055-9069-1e59a9c4e392-workload-socket\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-cjmnk\" (UID: \"5359ba6b-7deb-4055-9069-1e59a9c4e392\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-cjmnk" Oct 30 21:53:24.544221 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:24.543582 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5359ba6b-7deb-4055-9069-1e59a9c4e392-credential-socket\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-cjmnk\" (UID: \"5359ba6b-7deb-4055-9069-1e59a9c4e392\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-cjmnk" Oct 30 21:53:24.556544 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:24.556487 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5359ba6b-7deb-4055-9069-1e59a9c4e392-istio-envoy\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-cjmnk\" (UID: \"5359ba6b-7deb-4055-9069-1e59a9c4e392\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-cjmnk" Oct 30 21:53:24.567458 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:24.563915 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5359ba6b-7deb-4055-9069-1e59a9c4e392-istio-podinfo\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-cjmnk\" (UID: \"5359ba6b-7deb-4055-9069-1e59a9c4e392\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-cjmnk" Oct 30 21:53:24.581462 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:24.580907 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98jht\" (UniqueName: \"kubernetes.io/projected/5359ba6b-7deb-4055-9069-1e59a9c4e392-kube-api-access-98jht\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-cjmnk\" (UID: \"5359ba6b-7deb-4055-9069-1e59a9c4e392\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-cjmnk" Oct 30 21:53:24.581462 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:24.581345 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5359ba6b-7deb-4055-9069-1e59a9c4e392-istio-token\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-cjmnk\" (UID: \"5359ba6b-7deb-4055-9069-1e59a9c4e392\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-cjmnk" Oct 30 21:53:24.679230 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:24.679190 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:24.685878 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:24.685854 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-cjmnk" Oct 30 21:53:24.731330 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:24.731300 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-ceb36a/distinct-2-w3-ceb36a-kserve-7c59695467-46gdc"] Oct 30 21:53:24.983914 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:24.983824 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-2-w3-ceb36a-kserve-7c59695467-46gdc" event={"ID":"f1b6e92c-2952-4bcb-ae33-91e7271342fe","Type":"ContainerStarted","Data":"0db241e1eb057c733cc8bb28926f2e8cca1a3d419592d08dc7f68a9ebed706d8"} Oct 30 21:53:24.983914 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:24.983871 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-2-w3-ceb36a-kserve-7c59695467-46gdc" event={"ID":"f1b6e92c-2952-4bcb-ae33-91e7271342fe","Type":"ContainerStarted","Data":"36b31ec83a29ae7f045030a95327dc2936bee3b0d465f67e73dc356e79e1458a"} Oct 30 21:53:25.005911 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:25.005883 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-cjmnk"] Oct 30 21:53:25.007906 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:53:25.007877 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5359ba6b_7deb_4055_9069_1e59a9c4e392.slice/crio-ab1da138b72e84fedaf338b2f560eb00b40365b94582a76b0077ddb33bd732d4 WatchSource:0}: Error finding container ab1da138b72e84fedaf338b2f560eb00b40365b94582a76b0077ddb33bd732d4: Status 404 returned error can't find the container with id ab1da138b72e84fedaf338b2f560eb00b40365b94582a76b0077ddb33bd732d4 Oct 30 21:53:25.010291 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:25.010262 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:53:25.010373 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:25.010321 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:53:25.010373 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:25.010351 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:53:25.012162 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:25.012114 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-2-w3-ceb36a-kserve-7c59695467-46gdc" podStartSLOduration=2.01209913 podStartE2EDuration="2.01209913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:25.011709545 +0000 UTC m=+771.182674727" watchObservedRunningTime="2026-10-30 21:53:25.01209913 +0000 UTC m=+771.183064313" Oct 30 21:53:25.219889 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:25.219857 6720 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 21:53:25.220279 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:25.219918 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9" podUID="13913563-7222-407c-98b1-5c1a17d10b82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 21:53:25.385420 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:25.385341 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-2-w3-ceb36a-kserve-7c59695467-46gdc" Oct 30 21:53:25.584163 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:25.584127 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:26.031396 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:26.031310 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-cjmnk" event={"ID":"5359ba6b-7deb-4055-9069-1e59a9c4e392","Type":"ContainerStarted","Data":"da85a37615c3f04fa1c21365d780a1fa28e97123dc6dea03c41dffa8a75c4963"} Oct 30 21:53:26.031597 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:26.031437 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-cjmnk" event={"ID":"5359ba6b-7deb-4055-9069-1e59a9c4e392","Type":"ContainerStarted","Data":"ab1da138b72e84fedaf338b2f560eb00b40365b94582a76b0077ddb33bd732d4"} Oct 30 21:53:26.231403 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:26.219115 6720 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 21:53:26.231403 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:26.219184 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9" podUID="13913563-7222-407c-98b1-5c1a17d10b82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 21:53:26.315483 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:26.300136 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:26.695409 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:26.694781 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-cjmnk" Oct 30 21:53:26.699409 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:26.697541 6720 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-cjmnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 21:53:26.699409 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:26.697611 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-cjmnk" podUID="5359ba6b-7deb-4055-9069-1e59a9c4e392" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 21:53:27.236421 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:27.228764 6720 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 21:53:27.236421 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:27.228825 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9" podUID="13913563-7222-407c-98b1-5c1a17d10b82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 21:53:27.315894 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:27.315835 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:27.700419 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:27.696704 6720 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-cjmnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 21:53:27.700419 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:27.696763 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-cjmnk" podUID="5359ba6b-7deb-4055-9069-1e59a9c4e392" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 21:53:28.219234 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:28.219194 6720 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 21:53:28.219426 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:28.219247 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9" podUID="13913563-7222-407c-98b1-5c1a17d10b82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 21:53:28.322140 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:28.322105 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:28.691824 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:28.691789 6720 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-cjmnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 21:53:28.692138 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:28.692109 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-cjmnk" podUID="5359ba6b-7deb-4055-9069-1e59a9c4e392" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 21:53:29.223171 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:29.222738 6720 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 21:53:29.223171 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:29.222804 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9" podUID="13913563-7222-407c-98b1-5c1a17d10b82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 21:53:29.264423 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:29.262211 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:29.687238 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:29.687202 6720 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-cjmnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 21:53:29.687726 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:29.687264 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-cjmnk" podUID="5359ba6b-7deb-4055-9069-1e59a9c4e392" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 21:53:30.189410 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:30.188576 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:30.578144 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:30.576525 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9" Oct 30 21:53:30.578144 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:30.577493 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9" Oct 30 21:53:30.578576 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:30.578558 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9" Oct 30 21:53:30.610432 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:30.610349 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-cjmnk" podStartSLOduration=6.610331815 podStartE2EDuration="6.610331815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:26.082759996 +0000 UTC m=+772.253725192" watchObservedRunningTime="2026-10-30 21:53:30.610331815 +0000 UTC m=+776.781297001" Oct 30 21:53:30.687262 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:30.687222 6720 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-cjmnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 21:53:30.687941 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:30.687279 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-cjmnk" podUID="5359ba6b-7deb-4055-9069-1e59a9c4e392" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 21:53:31.043581 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:31.043547 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:31.206654 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:31.206618 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:31.572412 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:31.569940 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:31.687411 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:31.687345 6720 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-cjmnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 21:53:31.687809 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:31.687447 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-cjmnk" podUID="5359ba6b-7deb-4055-9069-1e59a9c4e392" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 21:53:32.437341 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:32.437290 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:32.636550 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:32.636505 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:32.792036 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:32.791929 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:33.150469 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:33.150440 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:33.727555 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:33.727512 6720 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-cjmnk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:53:33.727728 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:33.727591 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-cjmnk" podUID="5359ba6b-7deb-4055-9069-1e59a9c4e392" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:53:33.914711 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:33.914675 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-cjmnk" Oct 30 21:53:33.915189 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:33.915115 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-cjmnk" Oct 30 21:53:33.916067 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:33.916041 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-cjmnk" Oct 30 21:53:34.033660 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:34.033560 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:34.385147 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:34.385119 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-2-w3-ceb36a-kserve-7c59695467-46gdc" Oct 30 21:53:34.398519 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:34.398494 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-2-w3-ceb36a-kserve-7c59695467-46gdc" Oct 30 21:53:34.929012 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:34.928977 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:35.150277 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:35.150251 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-2-w3-ceb36a-kserve-7c59695467-46gdc" Oct 30 21:53:35.531404 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:35.531356 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:35.693942 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:35.693903 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:36.439969 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:36.439931 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:37.158821 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:37.158789 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:38.038890 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:38.038853 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:39.383867 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:39.383831 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:40.461017 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:53:40.460975 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 21:53:40.841581 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:40.841478 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/simulator-w4-939a16-kserve-74f845bf84-4c7b6" podUID="9bb6805f-f355-4ea8-80b7-542215c42022" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:53:41.614411 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:41.610816 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/distinct-w4-939a16-kserve-7f88f68c56-cngtk" podUID="85316c69-7ebd-4fb6-9195-44d1fba2dfbc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:53:41.832467 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:41.832423 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/simulator-w0-1529dc-kserve-8466d8d645-l54cf" podUID="257edb08-2e86-4f49-be57-848ace5958b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:53:41.842894 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:41.842850 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/simulator-w4-939a16-kserve-74f845bf84-4c7b6" podUID="9bb6805f-f355-4ea8-80b7-542215c42022" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:53:41.844153 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:41.844132 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-939a16/simulator-w4-939a16-kserve-74f845bf84-4c7b6" Oct 30 21:53:41.896819 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:41.896641 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-939a16/simulator-w4-939a16-kserve-74f845bf84-4c7b6" Oct 30 21:53:42.162466 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:42.155472 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-2-w3-ceb36a-kserve-7c59695467-46gdc" podUID="f1b6e92c-2952-4bcb-ae33-91e7271342fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:53:42.450087 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:42.449990 6720 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1529dc/distinct-2-w0-1529dc-kserve-7788894c6-7mbrr" podUID="09662f19-1ef9-406c-85ef-e402505875b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:53:42.627430 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:42.627365 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:43.711653 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:43.711618 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:44.670964 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:44.670899 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:45.611865 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:45.611829 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:46.435732 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:46.435700 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:47.549759 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:47.549720 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:48.337235 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:48.337193 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:49.146280 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:49.146245 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:49.334250 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:49.334213 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:49.974145 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:49.974094 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:51.059290 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:51.059252 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:52.861440 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:52.860711 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:54.058788 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:54.058749 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:55.019873 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:55.019846 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:55.845995 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:55.845961 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:56.930327 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:56.930276 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:58.006942 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:58.006911 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:53:59.291666 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:53:59.291632 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:00.448472 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:00.448416 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:01.301620 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:01.301587 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:02.087406 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:02.087310 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:02.745358 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:02.745325 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:03.432159 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:03.432123 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:04.130512 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:04.130424 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:05.693950 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:05.693914 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:05.961336 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:05.961262 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-0f26c5/distinct-w1-0f26c5-kserve-7d6b79498b-lvsjw"] Oct 30 21:54:05.965680 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:05.965655 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-0f26c5/distinct-w1-0f26c5-kserve-7d6b79498b-lvsjw" Oct 30 21:54:05.978241 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:05.975925 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-0f26c5\"/\"default-dockercfg-ck8pq\"" Oct 30 21:54:05.978241 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:05.977733 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-0f26c5/distinct-w1-0f26c5-kserve-7d6b79498b-lvsjw"] Oct 30 21:54:05.981694 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:05.981666 6720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-0f26c5\"/\"openshift-service-ca.crt\"" Oct 30 21:54:05.981865 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:05.981847 6720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-0f26c5\"/\"kube-root-ca.crt\"" Oct 30 21:54:06.054684 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:06.054636 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/457d2f97-63ab-41f4-b761-294d0ba3045e-tls-certs\") pod \"distinct-w1-0f26c5-kserve-7d6b79498b-lvsjw\" (UID: \"457d2f97-63ab-41f4-b761-294d0ba3045e\") " pod="e2e-models-e2e-worker-w1-0f26c5/distinct-w1-0f26c5-kserve-7d6b79498b-lvsjw" Oct 30 21:54:06.054684 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:06.054682 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qckh\" (UniqueName: \"kubernetes.io/projected/457d2f97-63ab-41f4-b761-294d0ba3045e-kube-api-access-4qckh\") pod \"distinct-w1-0f26c5-kserve-7d6b79498b-lvsjw\" (UID: \"457d2f97-63ab-41f4-b761-294d0ba3045e\") " pod="e2e-models-e2e-worker-w1-0f26c5/distinct-w1-0f26c5-kserve-7d6b79498b-lvsjw" Oct 30 21:54:06.054943 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:06.054745 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/457d2f97-63ab-41f4-b761-294d0ba3045e-home\") pod \"distinct-w1-0f26c5-kserve-7d6b79498b-lvsjw\" (UID: \"457d2f97-63ab-41f4-b761-294d0ba3045e\") " pod="e2e-models-e2e-worker-w1-0f26c5/distinct-w1-0f26c5-kserve-7d6b79498b-lvsjw" Oct 30 21:54:06.054943 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:06.054783 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/457d2f97-63ab-41f4-b761-294d0ba3045e-model-cache\") pod \"distinct-w1-0f26c5-kserve-7d6b79498b-lvsjw\" (UID: \"457d2f97-63ab-41f4-b761-294d0ba3045e\") " pod="e2e-models-e2e-worker-w1-0f26c5/distinct-w1-0f26c5-kserve-7d6b79498b-lvsjw" Oct 30 21:54:06.055054 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:06.054950 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/457d2f97-63ab-41f4-b761-294d0ba3045e-tmp-dir\") pod \"distinct-w1-0f26c5-kserve-7d6b79498b-lvsjw\" (UID: \"457d2f97-63ab-41f4-b761-294d0ba3045e\") " pod="e2e-models-e2e-worker-w1-0f26c5/distinct-w1-0f26c5-kserve-7d6b79498b-lvsjw" Oct 30 21:54:06.055054 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:06.055008 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/457d2f97-63ab-41f4-b761-294d0ba3045e-dshm\") pod \"distinct-w1-0f26c5-kserve-7d6b79498b-lvsjw\" (UID: \"457d2f97-63ab-41f4-b761-294d0ba3045e\") " pod="e2e-models-e2e-worker-w1-0f26c5/distinct-w1-0f26c5-kserve-7d6b79498b-lvsjw" Oct 30 21:54:06.068152 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:06.068096 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:06.081929 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:06.081908 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-0f26c5\"/\"distinct-w1-0f26c5-kserve-self-signed-certs\"" Oct 30 21:54:06.156107 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:06.156058 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/457d2f97-63ab-41f4-b761-294d0ba3045e-dshm\") pod \"distinct-w1-0f26c5-kserve-7d6b79498b-lvsjw\" (UID: \"457d2f97-63ab-41f4-b761-294d0ba3045e\") " pod="e2e-models-e2e-worker-w1-0f26c5/distinct-w1-0f26c5-kserve-7d6b79498b-lvsjw" Oct 30 21:54:06.156280 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:06.156189 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/457d2f97-63ab-41f4-b761-294d0ba3045e-tls-certs\") pod \"distinct-w1-0f26c5-kserve-7d6b79498b-lvsjw\" (UID: \"457d2f97-63ab-41f4-b761-294d0ba3045e\") " pod="e2e-models-e2e-worker-w1-0f26c5/distinct-w1-0f26c5-kserve-7d6b79498b-lvsjw" Oct 30 21:54:06.156280 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:06.156220 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qckh\" (UniqueName: \"kubernetes.io/projected/457d2f97-63ab-41f4-b761-294d0ba3045e-kube-api-access-4qckh\") pod \"distinct-w1-0f26c5-kserve-7d6b79498b-lvsjw\" (UID: \"457d2f97-63ab-41f4-b761-294d0ba3045e\") " pod="e2e-models-e2e-worker-w1-0f26c5/distinct-w1-0f26c5-kserve-7d6b79498b-lvsjw" Oct 30 21:54:06.156422 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:06.156283 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/457d2f97-63ab-41f4-b761-294d0ba3045e-home\") pod \"distinct-w1-0f26c5-kserve-7d6b79498b-lvsjw\" (UID: \"457d2f97-63ab-41f4-b761-294d0ba3045e\") " pod="e2e-models-e2e-worker-w1-0f26c5/distinct-w1-0f26c5-kserve-7d6b79498b-lvsjw" Oct 30 21:54:06.156422 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:06.156316 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/457d2f97-63ab-41f4-b761-294d0ba3045e-model-cache\") pod \"distinct-w1-0f26c5-kserve-7d6b79498b-lvsjw\" (UID: \"457d2f97-63ab-41f4-b761-294d0ba3045e\") " pod="e2e-models-e2e-worker-w1-0f26c5/distinct-w1-0f26c5-kserve-7d6b79498b-lvsjw" Oct 30 21:54:06.156534 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:06.156471 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/457d2f97-63ab-41f4-b761-294d0ba3045e-tmp-dir\") pod \"distinct-w1-0f26c5-kserve-7d6b79498b-lvsjw\" (UID: \"457d2f97-63ab-41f4-b761-294d0ba3045e\") " pod="e2e-models-e2e-worker-w1-0f26c5/distinct-w1-0f26c5-kserve-7d6b79498b-lvsjw" Oct 30 21:54:06.156927 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:06.156702 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/457d2f97-63ab-41f4-b761-294d0ba3045e-home\") pod \"distinct-w1-0f26c5-kserve-7d6b79498b-lvsjw\" (UID: \"457d2f97-63ab-41f4-b761-294d0ba3045e\") " pod="e2e-models-e2e-worker-w1-0f26c5/distinct-w1-0f26c5-kserve-7d6b79498b-lvsjw" Oct 30 21:54:06.156927 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:06.156800 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/457d2f97-63ab-41f4-b761-294d0ba3045e-tmp-dir\") pod \"distinct-w1-0f26c5-kserve-7d6b79498b-lvsjw\" (UID: \"457d2f97-63ab-41f4-b761-294d0ba3045e\") " pod="e2e-models-e2e-worker-w1-0f26c5/distinct-w1-0f26c5-kserve-7d6b79498b-lvsjw" Oct 30 21:54:06.156927 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:06.156871 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/457d2f97-63ab-41f4-b761-294d0ba3045e-model-cache\") pod \"distinct-w1-0f26c5-kserve-7d6b79498b-lvsjw\" (UID: \"457d2f97-63ab-41f4-b761-294d0ba3045e\") " pod="e2e-models-e2e-worker-w1-0f26c5/distinct-w1-0f26c5-kserve-7d6b79498b-lvsjw" Oct 30 21:54:06.159077 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:06.159054 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/457d2f97-63ab-41f4-b761-294d0ba3045e-tls-certs\") pod \"distinct-w1-0f26c5-kserve-7d6b79498b-lvsjw\" (UID: \"457d2f97-63ab-41f4-b761-294d0ba3045e\") " pod="e2e-models-e2e-worker-w1-0f26c5/distinct-w1-0f26c5-kserve-7d6b79498b-lvsjw" Oct 30 21:54:06.159203 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:06.159075 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/457d2f97-63ab-41f4-b761-294d0ba3045e-dshm\") pod \"distinct-w1-0f26c5-kserve-7d6b79498b-lvsjw\" (UID: \"457d2f97-63ab-41f4-b761-294d0ba3045e\") " pod="e2e-models-e2e-worker-w1-0f26c5/distinct-w1-0f26c5-kserve-7d6b79498b-lvsjw" Oct 30 21:54:06.172942 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:06.172894 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qckh\" (UniqueName: \"kubernetes.io/projected/457d2f97-63ab-41f4-b761-294d0ba3045e-kube-api-access-4qckh\") pod \"distinct-w1-0f26c5-kserve-7d6b79498b-lvsjw\" (UID: \"457d2f97-63ab-41f4-b761-294d0ba3045e\") " pod="e2e-models-e2e-worker-w1-0f26c5/distinct-w1-0f26c5-kserve-7d6b79498b-lvsjw" Oct 30 21:54:06.282456 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:06.281622 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-0f26c5/distinct-w1-0f26c5-kserve-7d6b79498b-lvsjw" Oct 30 21:54:06.695412 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:06.678868 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:06.710851 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:06.710827 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-0f26c5/distinct-w1-0f26c5-kserve-7d6b79498b-lvsjw"] Oct 30 21:54:07.057463 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:07.056131 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-ljm7z"] Oct 30 21:54:07.061204 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:07.061183 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-ljm7z" Oct 30 21:54:07.094615 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:07.082681 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-ljm7z"] Oct 30 21:54:07.267507 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:07.267415 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/919d285e-2e4f-41b9-95f8-87144df6a112-istio-token\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-ljm7z\" (UID: \"919d285e-2e4f-41b9-95f8-87144df6a112\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-ljm7z" Oct 30 21:54:07.271529 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:07.267822 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4cpl\" (UniqueName: \"kubernetes.io/projected/919d285e-2e4f-41b9-95f8-87144df6a112-kube-api-access-l4cpl\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-ljm7z\" (UID: \"919d285e-2e4f-41b9-95f8-87144df6a112\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-ljm7z" Oct 30 21:54:07.272075 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:07.272053 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/919d285e-2e4f-41b9-95f8-87144df6a112-istio-podinfo\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-ljm7z\" (UID: \"919d285e-2e4f-41b9-95f8-87144df6a112\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-ljm7z" Oct 30 21:54:07.303447 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:07.272244 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/919d285e-2e4f-41b9-95f8-87144df6a112-istio-data\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-ljm7z\" (UID: \"919d285e-2e4f-41b9-95f8-87144df6a112\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-ljm7z" Oct 30 21:54:07.303447 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:07.298481 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/919d285e-2e4f-41b9-95f8-87144df6a112-istio-envoy\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-ljm7z\" (UID: \"919d285e-2e4f-41b9-95f8-87144df6a112\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-ljm7z" Oct 30 21:54:07.303447 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:07.298551 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/919d285e-2e4f-41b9-95f8-87144df6a112-istiod-ca-cert\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-ljm7z\" (UID: \"919d285e-2e4f-41b9-95f8-87144df6a112\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-ljm7z" Oct 30 21:54:07.303447 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:07.298593 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/919d285e-2e4f-41b9-95f8-87144df6a112-workload-socket\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-ljm7z\" (UID: \"919d285e-2e4f-41b9-95f8-87144df6a112\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-ljm7z" Oct 30 21:54:07.303447 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:07.298628 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/919d285e-2e4f-41b9-95f8-87144df6a112-credential-socket\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-ljm7z\" (UID: \"919d285e-2e4f-41b9-95f8-87144df6a112\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-ljm7z" Oct 30 21:54:07.303447 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:07.298684 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/919d285e-2e4f-41b9-95f8-87144df6a112-workload-certs\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-ljm7z\" (UID: \"919d285e-2e4f-41b9-95f8-87144df6a112\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-ljm7z" Oct 30 21:54:07.380570 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:07.380421 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr"] Oct 30 21:54:07.420992 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:07.420956 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/919d285e-2e4f-41b9-95f8-87144df6a112-istio-podinfo\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-ljm7z\" (UID: \"919d285e-2e4f-41b9-95f8-87144df6a112\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-ljm7z" Oct 30 21:54:07.472756 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:07.472724 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/919d285e-2e4f-41b9-95f8-87144df6a112-istio-data\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-ljm7z\" (UID: \"919d285e-2e4f-41b9-95f8-87144df6a112\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-ljm7z" Oct 30 21:54:07.479715 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:07.479687 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/919d285e-2e4f-41b9-95f8-87144df6a112-istio-envoy\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-ljm7z\" (UID: \"919d285e-2e4f-41b9-95f8-87144df6a112\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-ljm7z" Oct 30 21:54:07.503116 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:07.493593 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/919d285e-2e4f-41b9-95f8-87144df6a112-istiod-ca-cert\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-ljm7z\" (UID: \"919d285e-2e4f-41b9-95f8-87144df6a112\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-ljm7z" Oct 30 21:54:07.503116 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:07.493648 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/919d285e-2e4f-41b9-95f8-87144df6a112-workload-socket\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-ljm7z\" (UID: \"919d285e-2e4f-41b9-95f8-87144df6a112\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-ljm7z" Oct 30 21:54:07.503116 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:07.493678 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/919d285e-2e4f-41b9-95f8-87144df6a112-credential-socket\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-ljm7z\" (UID: \"919d285e-2e4f-41b9-95f8-87144df6a112\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-ljm7z" Oct 30 21:54:07.503116 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:07.493733 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/919d285e-2e4f-41b9-95f8-87144df6a112-workload-certs\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-ljm7z\" (UID: \"919d285e-2e4f-41b9-95f8-87144df6a112\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-ljm7z" Oct 30 21:54:07.503116 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:07.493802 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/919d285e-2e4f-41b9-95f8-87144df6a112-istio-token\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-ljm7z\" (UID: \"919d285e-2e4f-41b9-95f8-87144df6a112\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-ljm7z" Oct 30 21:54:07.503116 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:07.493858 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4cpl\" (UniqueName: \"kubernetes.io/projected/919d285e-2e4f-41b9-95f8-87144df6a112-kube-api-access-l4cpl\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-ljm7z\" (UID: \"919d285e-2e4f-41b9-95f8-87144df6a112\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-ljm7z" Oct 30 21:54:07.503116 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:07.473338 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/919d285e-2e4f-41b9-95f8-87144df6a112-istio-data\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-ljm7z\" (UID: \"919d285e-2e4f-41b9-95f8-87144df6a112\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-ljm7z" Oct 30 21:54:07.503116 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:07.494829 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/919d285e-2e4f-41b9-95f8-87144df6a112-credential-socket\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-ljm7z\" (UID: \"919d285e-2e4f-41b9-95f8-87144df6a112\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-ljm7z" Oct 30 21:54:07.503116 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:07.449759 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/919d285e-2e4f-41b9-95f8-87144df6a112-istio-podinfo\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-ljm7z\" (UID: \"919d285e-2e4f-41b9-95f8-87144df6a112\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-ljm7z" Oct 30 21:54:07.503116 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:07.495085 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/919d285e-2e4f-41b9-95f8-87144df6a112-workload-socket\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-ljm7z\" (UID: \"919d285e-2e4f-41b9-95f8-87144df6a112\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-ljm7z" Oct 30 21:54:07.503116 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:07.495330 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/919d285e-2e4f-41b9-95f8-87144df6a112-workload-certs\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-ljm7z\" (UID: \"919d285e-2e4f-41b9-95f8-87144df6a112\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-ljm7z" Oct 30 21:54:07.512444 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:07.510169 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/919d285e-2e4f-41b9-95f8-87144df6a112-istiod-ca-cert\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-ljm7z\" (UID: \"919d285e-2e4f-41b9-95f8-87144df6a112\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-ljm7z" Oct 30 21:54:07.515056 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:07.515032 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr" Oct 30 21:54:07.567451 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:07.563983 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-0f26c5/distinct-w1-0f26c5-kserve-7d6b79498b-lvsjw" event={"ID":"457d2f97-63ab-41f4-b761-294d0ba3045e","Type":"ContainerStarted","Data":"e3beec726f3388452ebce7cdd62b42b8a234745f8d7cdf1dc83f6216d7dcf25a"} Oct 30 21:54:07.567451 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:07.565340 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-0f26c5/distinct-w1-0f26c5-kserve-7d6b79498b-lvsjw" event={"ID":"457d2f97-63ab-41f4-b761-294d0ba3045e","Type":"ContainerStarted","Data":"1fb3f2dd4f80080bd6e042321b4bd014a921f060a2adac73a3edbc7cd51e292f"} Oct 30 21:54:07.567724 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:07.567701 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr"] Oct 30 21:54:07.587471 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:07.580148 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4cpl\" (UniqueName: \"kubernetes.io/projected/919d285e-2e4f-41b9-95f8-87144df6a112-kube-api-access-l4cpl\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-ljm7z\" (UID: \"919d285e-2e4f-41b9-95f8-87144df6a112\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-ljm7z" Oct 30 21:54:07.591928 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:07.591900 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/919d285e-2e4f-41b9-95f8-87144df6a112-istio-token\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-ljm7z\" (UID: \"919d285e-2e4f-41b9-95f8-87144df6a112\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-ljm7z" Oct 30 21:54:07.605487 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:07.592534 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/919d285e-2e4f-41b9-95f8-87144df6a112-istio-envoy\") pod \"e2e-worker-w0-1529dc-openshift-default-5d54c4f69-ljm7z\" (UID: \"919d285e-2e4f-41b9-95f8-87144df6a112\") " pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-ljm7z" Oct 30 21:54:07.605682 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:07.592813 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-0f26c5/distinct-w1-0f26c5-kserve-7d6b79498b-lvsjw" podStartSLOduration=2.592794883 podStartE2EDuration="2.592794883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:07.592725131 +0000 UTC m=+813.763690315" watchObservedRunningTime="2026-10-30 21:54:07.592794883 +0000 UTC m=+813.763760066" Oct 30 21:54:07.617408 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:07.607115 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35de2b6b-4247-488f-baa1-182d56e82253-istio-podinfo\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr\" (UID: \"35de2b6b-4247-488f-baa1-182d56e82253\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr" Oct 30 21:54:07.618156 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:07.618123 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35de2b6b-4247-488f-baa1-182d56e82253-istio-token\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr\" (UID: \"35de2b6b-4247-488f-baa1-182d56e82253\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr" Oct 30 21:54:07.618363 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:07.618346 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjt45\" (UniqueName: \"kubernetes.io/projected/35de2b6b-4247-488f-baa1-182d56e82253-kube-api-access-fjt45\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr\" (UID: \"35de2b6b-4247-488f-baa1-182d56e82253\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr" Oct 30 21:54:07.622669 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:07.622646 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35de2b6b-4247-488f-baa1-182d56e82253-istio-envoy\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr\" (UID: \"35de2b6b-4247-488f-baa1-182d56e82253\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr" Oct 30 21:54:07.622849 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:07.622833 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35de2b6b-4247-488f-baa1-182d56e82253-workload-socket\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr\" (UID: \"35de2b6b-4247-488f-baa1-182d56e82253\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr" Oct 30 21:54:07.623034 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:07.623019 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35de2b6b-4247-488f-baa1-182d56e82253-istiod-ca-cert\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr\" (UID: \"35de2b6b-4247-488f-baa1-182d56e82253\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr" Oct 30 21:54:07.630672 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:07.630651 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35de2b6b-4247-488f-baa1-182d56e82253-credential-socket\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr\" (UID: \"35de2b6b-4247-488f-baa1-182d56e82253\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr" Oct 30 21:54:07.631008 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:07.630989 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35de2b6b-4247-488f-baa1-182d56e82253-istio-data\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr\" (UID: \"35de2b6b-4247-488f-baa1-182d56e82253\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr" Oct 30 21:54:07.631203 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:07.631187 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35de2b6b-4247-488f-baa1-182d56e82253-workload-certs\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr\" (UID: \"35de2b6b-4247-488f-baa1-182d56e82253\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr" Oct 30 21:54:07.729636 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:07.729603 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-ljm7z" Oct 30 21:54:07.735019 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:07.734989 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35de2b6b-4247-488f-baa1-182d56e82253-istio-data\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr\" (UID: \"35de2b6b-4247-488f-baa1-182d56e82253\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr" Oct 30 21:54:07.735319 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:07.735301 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35de2b6b-4247-488f-baa1-182d56e82253-workload-certs\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr\" (UID: \"35de2b6b-4247-488f-baa1-182d56e82253\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr" Oct 30 21:54:07.735504 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:07.735489 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35de2b6b-4247-488f-baa1-182d56e82253-istio-podinfo\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr\" (UID: \"35de2b6b-4247-488f-baa1-182d56e82253\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr" Oct 30 21:54:07.815353 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:07.815317 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35de2b6b-4247-488f-baa1-182d56e82253-istio-token\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr\" (UID: \"35de2b6b-4247-488f-baa1-182d56e82253\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr" Oct 30 21:54:07.815564 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:07.815544 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjt45\" (UniqueName: \"kubernetes.io/projected/35de2b6b-4247-488f-baa1-182d56e82253-kube-api-access-fjt45\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr\" (UID: \"35de2b6b-4247-488f-baa1-182d56e82253\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr" Oct 30 21:54:07.868587 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:07.857534 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35de2b6b-4247-488f-baa1-182d56e82253-istio-envoy\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr\" (UID: \"35de2b6b-4247-488f-baa1-182d56e82253\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr" Oct 30 21:54:07.868587 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:07.857592 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35de2b6b-4247-488f-baa1-182d56e82253-workload-socket\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr\" (UID: \"35de2b6b-4247-488f-baa1-182d56e82253\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr" Oct 30 21:54:07.868587 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:07.857630 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35de2b6b-4247-488f-baa1-182d56e82253-istiod-ca-cert\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr\" (UID: \"35de2b6b-4247-488f-baa1-182d56e82253\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr" Oct 30 21:54:07.868587 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:07.857697 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35de2b6b-4247-488f-baa1-182d56e82253-credential-socket\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr\" (UID: \"35de2b6b-4247-488f-baa1-182d56e82253\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr" Oct 30 21:54:07.868587 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:07.858115 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35de2b6b-4247-488f-baa1-182d56e82253-credential-socket\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr\" (UID: \"35de2b6b-4247-488f-baa1-182d56e82253\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr" Oct 30 21:54:07.868587 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:07.756816 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35de2b6b-4247-488f-baa1-182d56e82253-istio-data\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr\" (UID: \"35de2b6b-4247-488f-baa1-182d56e82253\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr" Oct 30 21:54:07.868587 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:07.858733 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35de2b6b-4247-488f-baa1-182d56e82253-workload-socket\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr\" (UID: \"35de2b6b-4247-488f-baa1-182d56e82253\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr" Oct 30 21:54:07.868587 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:07.757051 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35de2b6b-4247-488f-baa1-182d56e82253-workload-certs\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr\" (UID: \"35de2b6b-4247-488f-baa1-182d56e82253\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr" Oct 30 21:54:07.868587 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:07.859350 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35de2b6b-4247-488f-baa1-182d56e82253-istiod-ca-cert\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr\" (UID: \"35de2b6b-4247-488f-baa1-182d56e82253\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr" Oct 30 21:54:07.931411 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:07.920216 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjt45\" (UniqueName: \"kubernetes.io/projected/35de2b6b-4247-488f-baa1-182d56e82253-kube-api-access-fjt45\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr\" (UID: \"35de2b6b-4247-488f-baa1-182d56e82253\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr" Oct 30 21:54:07.956413 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:07.920226 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35de2b6b-4247-488f-baa1-182d56e82253-istio-podinfo\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr\" (UID: \"35de2b6b-4247-488f-baa1-182d56e82253\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr" Oct 30 21:54:07.956834 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:07.930796 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35de2b6b-4247-488f-baa1-182d56e82253-istio-token\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr\" (UID: \"35de2b6b-4247-488f-baa1-182d56e82253\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr" Oct 30 21:54:07.978099 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:07.977527 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/premium-w3-ceb36a-kserve-6576bb476f-j5kdr" podUID="d065e788-c972-4683-84ed-86a70ed4791d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:08.006136 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:08.002461 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35de2b6b-4247-488f-baa1-182d56e82253-istio-envoy\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr\" (UID: \"35de2b6b-4247-488f-baa1-182d56e82253\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr" Oct 30 21:54:08.201459 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:08.176599 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-2-w3-ceb36a-kserve-7c59695467-46gdc" podUID="f1b6e92c-2952-4bcb-ae33-91e7271342fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 30 21:54:08.267212 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:08.267125 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr" Oct 30 21:54:08.313425 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:08.311576 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0f26c5/distinct-w1-0f26c5-kserve-7d6b79498b-lvsjw" Oct 30 21:54:08.563099 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:08.561125 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:54:08.563099 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:08.561281 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:54:08.563099 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:08.561402 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:54:08.581452 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:08.563851 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-ljm7z"] Oct 30 21:54:08.664943 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:08.649592 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/distinct-2-w0-1529dc-kserve-7788894c6-7mbrr" podUID="09662f19-1ef9-406c-85ef-e402505875b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:08.770410 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:08.765691 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-ljm7z" event={"ID":"919d285e-2e4f-41b9-95f8-87144df6a112","Type":"ContainerStarted","Data":"4b60e0ff5e2ab7539479ba3f95edddb1aa044b81c572806f0aa58c619f3850b9"} Oct 30 21:54:09.006476 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:09.005453 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr"] Oct 30 21:54:09.096450 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:09.069627 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:54:09.096450 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:09.069726 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:54:09.096450 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:09.069769 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:54:09.224406 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:09.222258 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:09.563505 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:09.563424 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-nv8nz"] Oct 30 21:54:09.585525 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:09.585495 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-nv8nz" Oct 30 21:54:09.610263 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:09.610234 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-nv8nz"] Oct 30 21:54:09.747085 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:09.747047 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/852dbcb8-3c2d-4622-beeb-3fc265a7f906-istiod-ca-cert\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-nv8nz\" (UID: \"852dbcb8-3c2d-4622-beeb-3fc265a7f906\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-nv8nz" Oct 30 21:54:09.747280 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:09.747268 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2lf4\" (UniqueName: \"kubernetes.io/projected/852dbcb8-3c2d-4622-beeb-3fc265a7f906-kube-api-access-v2lf4\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-nv8nz\" (UID: \"852dbcb8-3c2d-4622-beeb-3fc265a7f906\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-nv8nz" Oct 30 21:54:09.747409 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:09.747376 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/852dbcb8-3c2d-4622-beeb-3fc265a7f906-istio-token\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-nv8nz\" (UID: \"852dbcb8-3c2d-4622-beeb-3fc265a7f906\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-nv8nz" Oct 30 21:54:09.747513 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:09.747503 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/852dbcb8-3c2d-4622-beeb-3fc265a7f906-istio-envoy\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-nv8nz\" (UID: \"852dbcb8-3c2d-4622-beeb-3fc265a7f906\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-nv8nz" Oct 30 21:54:09.747582 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:09.747573 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/852dbcb8-3c2d-4622-beeb-3fc265a7f906-istio-data\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-nv8nz\" (UID: \"852dbcb8-3c2d-4622-beeb-3fc265a7f906\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-nv8nz" Oct 30 21:54:09.747676 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:09.747666 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/852dbcb8-3c2d-4622-beeb-3fc265a7f906-credential-socket\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-nv8nz\" (UID: \"852dbcb8-3c2d-4622-beeb-3fc265a7f906\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-nv8nz" Oct 30 21:54:09.747750 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:09.747741 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/852dbcb8-3c2d-4622-beeb-3fc265a7f906-istio-podinfo\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-nv8nz\" (UID: \"852dbcb8-3c2d-4622-beeb-3fc265a7f906\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-nv8nz" Oct 30 21:54:09.747825 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:09.747817 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/852dbcb8-3c2d-4622-beeb-3fc265a7f906-workload-socket\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-nv8nz\" (UID: \"852dbcb8-3c2d-4622-beeb-3fc265a7f906\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-nv8nz" Oct 30 21:54:09.747889 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:09.747880 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/852dbcb8-3c2d-4622-beeb-3fc265a7f906-workload-certs\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-nv8nz\" (UID: \"852dbcb8-3c2d-4622-beeb-3fc265a7f906\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-nv8nz" Oct 30 21:54:09.772855 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:09.772813 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr" event={"ID":"35de2b6b-4247-488f-baa1-182d56e82253","Type":"ContainerStarted","Data":"7b8aa45e3b0fe1435e9a7258dac91ffef05874b6c03fce67a56ea0869d3cd7fa"} Oct 30 21:54:09.773681 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:09.772876 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr" event={"ID":"35de2b6b-4247-488f-baa1-182d56e82253","Type":"ContainerStarted","Data":"dccbccc892c1ed30e8b7af8c7f2bd29bc99514cbab50ebbd6ec7002b17b45eda"} Oct 30 21:54:09.775357 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:09.775327 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-ljm7z" event={"ID":"919d285e-2e4f-41b9-95f8-87144df6a112","Type":"ContainerStarted","Data":"6b4e8d5d4fab470b21e36969ed58fe803ba5ccd055d4dd7ef8ceee78bdfe968c"} Oct 30 21:54:09.798138 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:09.798086 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr" podStartSLOduration=2.798074025 podStartE2EDuration="2.798074025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:09.797594497 +0000 UTC m=+815.968559680" watchObservedRunningTime="2026-10-30 21:54:09.798074025 +0000 UTC m=+815.969039206" Oct 30 21:54:09.837202 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:09.837098 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-ljm7z" podStartSLOduration=2.837080654 podStartE2EDuration="2.837080654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:09.835720243 +0000 UTC m=+816.006685427" watchObservedRunningTime="2026-10-30 21:54:09.837080654 +0000 UTC m=+816.008045839" Oct 30 21:54:09.849361 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:09.849319 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/852dbcb8-3c2d-4622-beeb-3fc265a7f906-workload-certs\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-nv8nz\" (UID: \"852dbcb8-3c2d-4622-beeb-3fc265a7f906\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-nv8nz" Oct 30 21:54:09.849513 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:09.849459 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/852dbcb8-3c2d-4622-beeb-3fc265a7f906-istiod-ca-cert\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-nv8nz\" (UID: \"852dbcb8-3c2d-4622-beeb-3fc265a7f906\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-nv8nz" Oct 30 21:54:09.849593 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:09.849515 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2lf4\" (UniqueName: \"kubernetes.io/projected/852dbcb8-3c2d-4622-beeb-3fc265a7f906-kube-api-access-v2lf4\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-nv8nz\" (UID: \"852dbcb8-3c2d-4622-beeb-3fc265a7f906\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-nv8nz" Oct 30 21:54:09.851506 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:09.849693 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/852dbcb8-3c2d-4622-beeb-3fc265a7f906-istio-token\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-nv8nz\" (UID: \"852dbcb8-3c2d-4622-beeb-3fc265a7f906\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-nv8nz" Oct 30 21:54:09.851506 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:09.850073 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/852dbcb8-3c2d-4622-beeb-3fc265a7f906-istiod-ca-cert\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-nv8nz\" (UID: \"852dbcb8-3c2d-4622-beeb-3fc265a7f906\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-nv8nz" Oct 30 21:54:09.851506 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:09.850129 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/852dbcb8-3c2d-4622-beeb-3fc265a7f906-istio-envoy\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-nv8nz\" (UID: \"852dbcb8-3c2d-4622-beeb-3fc265a7f906\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-nv8nz" Oct 30 21:54:09.851506 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:09.850162 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/852dbcb8-3c2d-4622-beeb-3fc265a7f906-istio-data\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-nv8nz\" (UID: \"852dbcb8-3c2d-4622-beeb-3fc265a7f906\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-nv8nz" Oct 30 21:54:09.851506 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:09.849779 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/852dbcb8-3c2d-4622-beeb-3fc265a7f906-workload-certs\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-nv8nz\" (UID: \"852dbcb8-3c2d-4622-beeb-3fc265a7f906\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-nv8nz" Oct 30 21:54:09.851506 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:09.850254 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/852dbcb8-3c2d-4622-beeb-3fc265a7f906-credential-socket\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-nv8nz\" (UID: \"852dbcb8-3c2d-4622-beeb-3fc265a7f906\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-nv8nz" Oct 30 21:54:09.851506 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:09.850303 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/852dbcb8-3c2d-4622-beeb-3fc265a7f906-istio-podinfo\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-nv8nz\" (UID: \"852dbcb8-3c2d-4622-beeb-3fc265a7f906\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-nv8nz" Oct 30 21:54:09.851506 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:09.850363 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/852dbcb8-3c2d-4622-beeb-3fc265a7f906-workload-socket\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-nv8nz\" (UID: \"852dbcb8-3c2d-4622-beeb-3fc265a7f906\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-nv8nz" Oct 30 21:54:09.851506 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:09.850663 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/852dbcb8-3c2d-4622-beeb-3fc265a7f906-istio-data\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-nv8nz\" (UID: \"852dbcb8-3c2d-4622-beeb-3fc265a7f906\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-nv8nz" Oct 30 21:54:09.851506 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:09.850949 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/852dbcb8-3c2d-4622-beeb-3fc265a7f906-credential-socket\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-nv8nz\" (UID: \"852dbcb8-3c2d-4622-beeb-3fc265a7f906\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-nv8nz" Oct 30 21:54:09.851506 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:09.850950 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/852dbcb8-3c2d-4622-beeb-3fc265a7f906-workload-socket\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-nv8nz\" (UID: \"852dbcb8-3c2d-4622-beeb-3fc265a7f906\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-nv8nz" Oct 30 21:54:09.852628 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:09.852605 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/852dbcb8-3c2d-4622-beeb-3fc265a7f906-istio-envoy\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-nv8nz\" (UID: \"852dbcb8-3c2d-4622-beeb-3fc265a7f906\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-nv8nz" Oct 30 21:54:09.853128 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:09.853106 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/852dbcb8-3c2d-4622-beeb-3fc265a7f906-istio-podinfo\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-nv8nz\" (UID: \"852dbcb8-3c2d-4622-beeb-3fc265a7f906\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-nv8nz" Oct 30 21:54:09.864623 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:09.864594 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/852dbcb8-3c2d-4622-beeb-3fc265a7f906-istio-token\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-nv8nz\" (UID: \"852dbcb8-3c2d-4622-beeb-3fc265a7f906\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-nv8nz" Oct 30 21:54:09.864772 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:09.864750 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2lf4\" (UniqueName: \"kubernetes.io/projected/852dbcb8-3c2d-4622-beeb-3fc265a7f906-kube-api-access-v2lf4\") pod \"e2e-worker-w4-939a16-openshift-default-5fccf878b7-nv8nz\" (UID: \"852dbcb8-3c2d-4622-beeb-3fc265a7f906\") " pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-nv8nz" Oct 30 21:54:09.925686 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:09.925648 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-nv8nz" Oct 30 21:54:10.271192 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:10.267769 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr" Oct 30 21:54:10.288530 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:10.271360 6720 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 21:54:10.288530 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:10.280500 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr" podUID="35de2b6b-4247-488f-baa1-182d56e82253" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 21:54:10.320590 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:10.313923 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-nv8nz"] Oct 30 21:54:10.321651 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:10.321520 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:10.322099 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:10.322057 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:54:10.322188 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:10.322167 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:54:10.322258 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:10.322212 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:54:10.514045 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:54:10.514007 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 21:54:10.731258 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:10.731223 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-ljm7z" Oct 30 21:54:10.732547 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:10.732519 6720 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-ljm7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 21:54:10.732707 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:10.732570 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-ljm7z" podUID="919d285e-2e4f-41b9-95f8-87144df6a112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 21:54:10.809354 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:10.809323 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-nv8nz" event={"ID":"852dbcb8-3c2d-4622-beeb-3fc265a7f906","Type":"ContainerStarted","Data":"dffc857ed3bc39215ce2a2834d0d81edb8861fe1f13dc1b926ef70b2246d4ae5"} Oct 30 21:54:10.809828 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:10.809799 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-nv8nz" event={"ID":"852dbcb8-3c2d-4622-beeb-3fc265a7f906","Type":"ContainerStarted","Data":"4a39a29a8f3e597f3a7215dd79770ee23e01297f6f915102cfdbf1c11717653b"} Oct 30 21:54:10.839879 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:10.839822 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-nv8nz" podStartSLOduration=1.839806683 podStartE2EDuration="1.839806683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:10.83781719 +0000 UTC m=+817.008782372" watchObservedRunningTime="2026-10-30 21:54:10.839806683 +0000 UTC m=+817.010771864" Oct 30 21:54:10.926716 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:10.926685 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-nv8nz" Oct 30 21:54:11.285410 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:11.284499 6720 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 21:54:11.285410 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:11.284565 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr" podUID="35de2b6b-4247-488f-baa1-182d56e82253" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 21:54:11.641286 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:11.640932 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:11.741748 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:11.741711 6720 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-ljm7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 21:54:11.742307 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:11.742273 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-ljm7z" podUID="919d285e-2e4f-41b9-95f8-87144df6a112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 21:54:11.933462 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:11.933200 6720 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-nv8nz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 21:54:11.933462 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:11.933255 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-nv8nz" podUID="852dbcb8-3c2d-4622-beeb-3fc265a7f906" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 21:54:12.274465 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:12.270834 6720 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 21:54:12.274465 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:12.270909 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr" podUID="35de2b6b-4247-488f-baa1-182d56e82253" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 21:54:12.732844 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:12.732802 6720 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-ljm7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 21:54:12.733114 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:12.733084 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-ljm7z" podUID="919d285e-2e4f-41b9-95f8-87144df6a112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 21:54:12.930239 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:12.929798 6720 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-nv8nz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 21:54:12.930239 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:12.929864 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-nv8nz" podUID="852dbcb8-3c2d-4622-beeb-3fc265a7f906" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 21:54:12.944459 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:12.944314 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:13.268412 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:13.267814 6720 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 21:54:13.268412 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:13.267875 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr" podUID="35de2b6b-4247-488f-baa1-182d56e82253" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 21:54:13.733411 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:13.732808 6720 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-ljm7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 21:54:13.733411 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:13.732886 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-ljm7z" podUID="919d285e-2e4f-41b9-95f8-87144df6a112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 21:54:13.927412 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:13.926770 6720 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-nv8nz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 21:54:13.927412 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:13.926837 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-nv8nz" podUID="852dbcb8-3c2d-4622-beeb-3fc265a7f906" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 21:54:14.049498 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:14.044954 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:14.276413 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:14.274078 6720 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 21:54:14.276413 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:14.274138 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr" podUID="35de2b6b-4247-488f-baa1-182d56e82253" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 21:54:14.733412 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:14.730752 6720 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-ljm7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 21:54:14.733412 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:14.730819 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-ljm7z" podUID="919d285e-2e4f-41b9-95f8-87144df6a112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 21:54:14.926824 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:14.926784 6720 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-nv8nz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 21:54:14.926998 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:14.926847 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-nv8nz" podUID="852dbcb8-3c2d-4622-beeb-3fc265a7f906" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 21:54:14.943519 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:14.943471 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/premium-w3-ceb36a-kserve-6576bb476f-j5kdr" podUID="d065e788-c972-4683-84ed-86a70ed4791d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:15.280410 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:15.276264 6720 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 21:54:15.280410 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:15.276323 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr" podUID="35de2b6b-4247-488f-baa1-182d56e82253" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 21:54:15.350033 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:15.349886 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:15.457407 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:15.456484 6720 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-2-w3-ceb36a-kserve-7c59695467-46gdc" podUID="f1b6e92c-2952-4bcb-ae33-91e7271342fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": context deadline exceeded" Oct 30 21:54:15.734595 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:15.734554 6720 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-ljm7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 21:54:15.734829 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:15.734618 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-ljm7z" podUID="919d285e-2e4f-41b9-95f8-87144df6a112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 21:54:15.927094 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:15.927046 6720 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-nv8nz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 21:54:15.927323 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:15.927120 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-nv8nz" podUID="852dbcb8-3c2d-4622-beeb-3fc265a7f906" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 21:54:16.269532 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:16.268723 6720 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 21:54:16.269532 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:16.268782 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr" podUID="35de2b6b-4247-488f-baa1-182d56e82253" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 21:54:16.283035 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:16.282182 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-0f26c5/distinct-w1-0f26c5-kserve-7d6b79498b-lvsjw" Oct 30 21:54:16.296793 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:16.296770 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-0f26c5/distinct-w1-0f26c5-kserve-7d6b79498b-lvsjw" Oct 30 21:54:16.335115 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:16.335082 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:16.731507 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:16.731472 6720 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-ljm7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 21:54:16.731712 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:16.731531 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-ljm7z" podUID="919d285e-2e4f-41b9-95f8-87144df6a112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 21:54:16.902430 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:16.902365 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0f26c5/distinct-w1-0f26c5-kserve-7d6b79498b-lvsjw" Oct 30 21:54:16.926128 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:16.926095 6720 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-nv8nz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 21:54:16.926277 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:16.926156 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-nv8nz" podUID="852dbcb8-3c2d-4622-beeb-3fc265a7f906" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 21:54:17.607409 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:17.607225 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:17.735688 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:17.734813 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr" Oct 30 21:54:17.736110 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:17.736089 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr" Oct 30 21:54:17.736684 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:17.736662 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-ljm7z" Oct 30 21:54:17.736889 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:17.736869 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr" Oct 30 21:54:17.737127 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:17.737104 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-ljm7z" Oct 30 21:54:17.738496 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:17.737991 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-ljm7z" Oct 30 21:54:18.338606 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:18.338571 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-nv8nz" Oct 30 21:54:18.339025 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:18.338996 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-nv8nz" Oct 30 21:54:18.339975 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:18.339951 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-nv8nz" Oct 30 21:54:18.699139 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:18.699104 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:19.626150 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:19.626075 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:20.968313 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:20.968276 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:22.498412 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:22.497531 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd"] Oct 30 21:54:22.518413 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:22.517655 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd" Oct 30 21:54:22.565414 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:22.562652 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd"] Oct 30 21:54:22.598412 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:22.598160 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1817b106-b8fd-4989-977b-115eca7245ec-credential-socket\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd\" (UID: \"1817b106-b8fd-4989-977b-115eca7245ec\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd" Oct 30 21:54:22.598412 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:22.598251 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1817b106-b8fd-4989-977b-115eca7245ec-istio-token\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd\" (UID: \"1817b106-b8fd-4989-977b-115eca7245ec\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd" Oct 30 21:54:22.598412 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:22.598298 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1817b106-b8fd-4989-977b-115eca7245ec-istio-envoy\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd\" (UID: \"1817b106-b8fd-4989-977b-115eca7245ec\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd" Oct 30 21:54:22.598412 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:22.598399 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1817b106-b8fd-4989-977b-115eca7245ec-istio-podinfo\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd\" (UID: \"1817b106-b8fd-4989-977b-115eca7245ec\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd" Oct 30 21:54:22.598802 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:22.598431 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1817b106-b8fd-4989-977b-115eca7245ec-workload-certs\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd\" (UID: \"1817b106-b8fd-4989-977b-115eca7245ec\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd" Oct 30 21:54:22.598802 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:22.598627 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv5pp\" (UniqueName: \"kubernetes.io/projected/1817b106-b8fd-4989-977b-115eca7245ec-kube-api-access-nv5pp\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd\" (UID: \"1817b106-b8fd-4989-977b-115eca7245ec\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd" Oct 30 21:54:22.598802 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:22.598705 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1817b106-b8fd-4989-977b-115eca7245ec-workload-socket\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd\" (UID: \"1817b106-b8fd-4989-977b-115eca7245ec\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd" Oct 30 21:54:22.598802 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:22.598749 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1817b106-b8fd-4989-977b-115eca7245ec-istiod-ca-cert\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd\" (UID: \"1817b106-b8fd-4989-977b-115eca7245ec\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd" Oct 30 21:54:22.599029 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:22.598835 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1817b106-b8fd-4989-977b-115eca7245ec-istio-data\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd\" (UID: \"1817b106-b8fd-4989-977b-115eca7245ec\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd" Oct 30 21:54:22.707410 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:22.707310 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv5pp\" (UniqueName: \"kubernetes.io/projected/1817b106-b8fd-4989-977b-115eca7245ec-kube-api-access-nv5pp\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd\" (UID: \"1817b106-b8fd-4989-977b-115eca7245ec\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd" Oct 30 21:54:22.707587 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:22.707422 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1817b106-b8fd-4989-977b-115eca7245ec-workload-socket\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd\" (UID: \"1817b106-b8fd-4989-977b-115eca7245ec\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd" Oct 30 21:54:22.707587 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:22.707451 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1817b106-b8fd-4989-977b-115eca7245ec-istiod-ca-cert\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd\" (UID: \"1817b106-b8fd-4989-977b-115eca7245ec\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd" Oct 30 21:54:22.707587 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:22.707532 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1817b106-b8fd-4989-977b-115eca7245ec-istio-data\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd\" (UID: \"1817b106-b8fd-4989-977b-115eca7245ec\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd" Oct 30 21:54:22.707779 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:22.707610 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1817b106-b8fd-4989-977b-115eca7245ec-credential-socket\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd\" (UID: \"1817b106-b8fd-4989-977b-115eca7245ec\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd" Oct 30 21:54:22.707779 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:22.707693 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1817b106-b8fd-4989-977b-115eca7245ec-istio-token\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd\" (UID: \"1817b106-b8fd-4989-977b-115eca7245ec\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd" Oct 30 21:54:22.707779 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:22.707750 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1817b106-b8fd-4989-977b-115eca7245ec-istio-envoy\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd\" (UID: \"1817b106-b8fd-4989-977b-115eca7245ec\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd" Oct 30 21:54:22.707940 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:22.707835 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1817b106-b8fd-4989-977b-115eca7245ec-istio-podinfo\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd\" (UID: \"1817b106-b8fd-4989-977b-115eca7245ec\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd" Oct 30 21:54:22.707940 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:22.707885 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1817b106-b8fd-4989-977b-115eca7245ec-workload-certs\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd\" (UID: \"1817b106-b8fd-4989-977b-115eca7245ec\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd" Oct 30 21:54:22.719421 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:22.714732 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1817b106-b8fd-4989-977b-115eca7245ec-workload-certs\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd\" (UID: \"1817b106-b8fd-4989-977b-115eca7245ec\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd" Oct 30 21:54:22.729413 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:22.727747 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1817b106-b8fd-4989-977b-115eca7245ec-istio-data\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd\" (UID: \"1817b106-b8fd-4989-977b-115eca7245ec\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd" Oct 30 21:54:22.758446 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:22.738959 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1817b106-b8fd-4989-977b-115eca7245ec-istiod-ca-cert\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd\" (UID: \"1817b106-b8fd-4989-977b-115eca7245ec\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd" Oct 30 21:54:22.758446 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:22.739569 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1817b106-b8fd-4989-977b-115eca7245ec-credential-socket\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd\" (UID: \"1817b106-b8fd-4989-977b-115eca7245ec\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd" Oct 30 21:54:22.782410 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:22.759878 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1817b106-b8fd-4989-977b-115eca7245ec-workload-socket\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd\" (UID: \"1817b106-b8fd-4989-977b-115eca7245ec\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd" Oct 30 21:54:22.824052 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:22.823979 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1817b106-b8fd-4989-977b-115eca7245ec-istio-token\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd\" (UID: \"1817b106-b8fd-4989-977b-115eca7245ec\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd" Oct 30 21:54:22.892995 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:22.892953 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1817b106-b8fd-4989-977b-115eca7245ec-istio-envoy\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd\" (UID: \"1817b106-b8fd-4989-977b-115eca7245ec\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd" Oct 30 21:54:22.914173 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:22.914141 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv5pp\" (UniqueName: \"kubernetes.io/projected/1817b106-b8fd-4989-977b-115eca7245ec-kube-api-access-nv5pp\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd\" (UID: \"1817b106-b8fd-4989-977b-115eca7245ec\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd" Oct 30 21:54:23.004489 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:23.004458 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-7v8vz"] Oct 30 21:54:23.038043 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:23.037965 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1817b106-b8fd-4989-977b-115eca7245ec-istio-podinfo\") pod \"e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd\" (UID: \"1817b106-b8fd-4989-977b-115eca7245ec\") " pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd" Oct 30 21:54:23.039591 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:23.039565 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-7v8vz"] Oct 30 21:54:23.039853 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:23.039838 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-7v8vz" Oct 30 21:54:23.169436 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:23.158734 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85e8605a-0cd7-4eaf-84e2-99e19e9ebebc-istio-podinfo\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-7v8vz\" (UID: \"85e8605a-0cd7-4eaf-84e2-99e19e9ebebc\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-7v8vz" Oct 30 21:54:23.216906 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:23.216868 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85e8605a-0cd7-4eaf-84e2-99e19e9ebebc-workload-certs\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-7v8vz\" (UID: \"85e8605a-0cd7-4eaf-84e2-99e19e9ebebc\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-7v8vz" Oct 30 21:54:23.217219 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:23.162733 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd" Oct 30 21:54:23.218107 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:23.217233 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85e8605a-0cd7-4eaf-84e2-99e19e9ebebc-istio-data\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-7v8vz\" (UID: \"85e8605a-0cd7-4eaf-84e2-99e19e9ebebc\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-7v8vz" Oct 30 21:54:23.218107 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:23.217595 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85e8605a-0cd7-4eaf-84e2-99e19e9ebebc-workload-socket\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-7v8vz\" (UID: \"85e8605a-0cd7-4eaf-84e2-99e19e9ebebc\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-7v8vz" Oct 30 21:54:23.218107 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:23.217653 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74wvk\" (UniqueName: \"kubernetes.io/projected/85e8605a-0cd7-4eaf-84e2-99e19e9ebebc-kube-api-access-74wvk\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-7v8vz\" (UID: \"85e8605a-0cd7-4eaf-84e2-99e19e9ebebc\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-7v8vz" Oct 30 21:54:23.218107 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:23.217689 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85e8605a-0cd7-4eaf-84e2-99e19e9ebebc-istio-token\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-7v8vz\" (UID: \"85e8605a-0cd7-4eaf-84e2-99e19e9ebebc\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-7v8vz" Oct 30 21:54:23.218107 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:23.217753 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85e8605a-0cd7-4eaf-84e2-99e19e9ebebc-istio-envoy\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-7v8vz\" (UID: \"85e8605a-0cd7-4eaf-84e2-99e19e9ebebc\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-7v8vz" Oct 30 21:54:23.218107 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:23.217827 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85e8605a-0cd7-4eaf-84e2-99e19e9ebebc-credential-socket\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-7v8vz\" (UID: \"85e8605a-0cd7-4eaf-84e2-99e19e9ebebc\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-7v8vz" Oct 30 21:54:23.218107 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:23.217873 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85e8605a-0cd7-4eaf-84e2-99e19e9ebebc-istiod-ca-cert\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-7v8vz\" (UID: \"85e8605a-0cd7-4eaf-84e2-99e19e9ebebc\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-7v8vz" Oct 30 21:54:23.335151 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:23.335067 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85e8605a-0cd7-4eaf-84e2-99e19e9ebebc-istiod-ca-cert\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-7v8vz\" (UID: \"85e8605a-0cd7-4eaf-84e2-99e19e9ebebc\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-7v8vz" Oct 30 21:54:23.335452 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:23.335413 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85e8605a-0cd7-4eaf-84e2-99e19e9ebebc-istio-podinfo\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-7v8vz\" (UID: \"85e8605a-0cd7-4eaf-84e2-99e19e9ebebc\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-7v8vz" Oct 30 21:54:23.344480 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:23.344452 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85e8605a-0cd7-4eaf-84e2-99e19e9ebebc-workload-certs\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-7v8vz\" (UID: \"85e8605a-0cd7-4eaf-84e2-99e19e9ebebc\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-7v8vz" Oct 30 21:54:23.344762 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:23.344738 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85e8605a-0cd7-4eaf-84e2-99e19e9ebebc-istio-data\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-7v8vz\" (UID: \"85e8605a-0cd7-4eaf-84e2-99e19e9ebebc\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-7v8vz" Oct 30 21:54:23.344911 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:23.344893 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85e8605a-0cd7-4eaf-84e2-99e19e9ebebc-workload-socket\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-7v8vz\" (UID: \"85e8605a-0cd7-4eaf-84e2-99e19e9ebebc\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-7v8vz" Oct 30 21:54:23.368437 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:23.365471 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74wvk\" (UniqueName: \"kubernetes.io/projected/85e8605a-0cd7-4eaf-84e2-99e19e9ebebc-kube-api-access-74wvk\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-7v8vz\" (UID: \"85e8605a-0cd7-4eaf-84e2-99e19e9ebebc\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-7v8vz" Oct 30 21:54:23.368437 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:23.365530 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85e8605a-0cd7-4eaf-84e2-99e19e9ebebc-istio-token\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-7v8vz\" (UID: \"85e8605a-0cd7-4eaf-84e2-99e19e9ebebc\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-7v8vz" Oct 30 21:54:23.368437 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:23.365600 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85e8605a-0cd7-4eaf-84e2-99e19e9ebebc-istio-envoy\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-7v8vz\" (UID: \"85e8605a-0cd7-4eaf-84e2-99e19e9ebebc\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-7v8vz" Oct 30 21:54:23.368437 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:23.365682 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85e8605a-0cd7-4eaf-84e2-99e19e9ebebc-credential-socket\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-7v8vz\" (UID: \"85e8605a-0cd7-4eaf-84e2-99e19e9ebebc\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-7v8vz" Oct 30 21:54:23.368437 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:23.366065 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85e8605a-0cd7-4eaf-84e2-99e19e9ebebc-credential-socket\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-7v8vz\" (UID: \"85e8605a-0cd7-4eaf-84e2-99e19e9ebebc\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-7v8vz" Oct 30 21:54:23.368437 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:23.345894 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85e8605a-0cd7-4eaf-84e2-99e19e9ebebc-istiod-ca-cert\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-7v8vz\" (UID: \"85e8605a-0cd7-4eaf-84e2-99e19e9ebebc\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-7v8vz" Oct 30 21:54:23.368437 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:23.346147 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85e8605a-0cd7-4eaf-84e2-99e19e9ebebc-workload-certs\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-7v8vz\" (UID: \"85e8605a-0cd7-4eaf-84e2-99e19e9ebebc\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-7v8vz" Oct 30 21:54:23.368437 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:23.346360 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85e8605a-0cd7-4eaf-84e2-99e19e9ebebc-istio-data\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-7v8vz\" (UID: \"85e8605a-0cd7-4eaf-84e2-99e19e9ebebc\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-7v8vz" Oct 30 21:54:23.368437 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:23.347074 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85e8605a-0cd7-4eaf-84e2-99e19e9ebebc-workload-socket\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-7v8vz\" (UID: \"85e8605a-0cd7-4eaf-84e2-99e19e9ebebc\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-7v8vz" Oct 30 21:54:23.387561 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:23.387532 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85e8605a-0cd7-4eaf-84e2-99e19e9ebebc-istio-podinfo\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-7v8vz\" (UID: \"85e8605a-0cd7-4eaf-84e2-99e19e9ebebc\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-7v8vz" Oct 30 21:54:23.435413 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:23.432826 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85e8605a-0cd7-4eaf-84e2-99e19e9ebebc-istio-envoy\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-7v8vz\" (UID: \"85e8605a-0cd7-4eaf-84e2-99e19e9ebebc\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-7v8vz" Oct 30 21:54:23.439151 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:23.439122 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74wvk\" (UniqueName: \"kubernetes.io/projected/85e8605a-0cd7-4eaf-84e2-99e19e9ebebc-kube-api-access-74wvk\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-7v8vz\" (UID: \"85e8605a-0cd7-4eaf-84e2-99e19e9ebebc\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-7v8vz" Oct 30 21:54:23.451409 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:23.450839 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85e8605a-0cd7-4eaf-84e2-99e19e9ebebc-istio-token\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-7v8vz\" (UID: \"85e8605a-0cd7-4eaf-84e2-99e19e9ebebc\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-7v8vz" Oct 30 21:54:23.569068 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:23.568778 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-7v8vz" Oct 30 21:54:23.659497 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:23.657815 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:23.945583 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:23.945554 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd"] Oct 30 21:54:23.947561 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:54:23.947523 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1817b106_b8fd_4989_977b_115eca7245ec.slice/crio-448525bab286267e657f7b9923b4d716a8e09d1d342b94ccd55571e5a44cc6fd WatchSource:0}: Error finding container 448525bab286267e657f7b9923b4d716a8e09d1d342b94ccd55571e5a44cc6fd: Status 404 returned error can't find the container with id 448525bab286267e657f7b9923b4d716a8e09d1d342b94ccd55571e5a44cc6fd Oct 30 21:54:23.950950 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:23.950913 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:54:23.951112 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:23.950989 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:54:23.951112 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:23.951024 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:54:24.057779 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:24.057738 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-7v8vz"] Oct 30 21:54:24.063615 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:24.063579 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd" event={"ID":"1817b106-b8fd-4989-977b-115eca7245ec","Type":"ContainerStarted","Data":"7e47218a45cb68904c8dfe0dfa9e739d9e105bfc58f1b8af7248e2ad6717aab3"} Oct 30 21:54:24.063822 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:24.063802 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd" event={"ID":"1817b106-b8fd-4989-977b-115eca7245ec","Type":"ContainerStarted","Data":"448525bab286267e657f7b9923b4d716a8e09d1d342b94ccd55571e5a44cc6fd"} Oct 30 21:54:24.068904 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:24.068181 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:54:24.068904 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:24.068262 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:54:24.068904 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:24.068304 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:54:24.163059 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:24.163035 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd" Oct 30 21:54:24.832127 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:24.832090 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:25.101144 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:25.101045 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-7v8vz" event={"ID":"85e8605a-0cd7-4eaf-84e2-99e19e9ebebc","Type":"ContainerStarted","Data":"81ddd0d3b4e6e80f74ae75ff158a5885e9abe291aee626002650b947fe949ccf"} Oct 30 21:54:25.101144 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:25.101102 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-7v8vz" event={"ID":"85e8605a-0cd7-4eaf-84e2-99e19e9ebebc","Type":"ContainerStarted","Data":"6248cd3550cd5b5f33e3483ef1fae06eaf8d4a7d0074b36604e13d1fc4afed70"} Oct 30 21:54:25.143237 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:25.143176 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-7v8vz" podStartSLOduration=3.143158005 podStartE2EDuration="3.143158005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:25.140319101 +0000 UTC m=+831.311284283" watchObservedRunningTime="2026-10-30 21:54:25.143158005 +0000 UTC m=+831.314123187" Oct 30 21:54:25.143523 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:25.143495 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd" podStartSLOduration=3.143486639 podStartE2EDuration="3.143486639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:24.120459483 +0000 UTC m=+830.291424670" watchObservedRunningTime="2026-10-30 21:54:25.143486639 +0000 UTC m=+831.314451820" Oct 30 21:54:25.186409 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:25.182276 6720 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 21:54:25.186409 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:25.182346 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd" podUID="1817b106-b8fd-4989-977b-115eca7245ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 21:54:25.577408 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:25.577087 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-7v8vz" Oct 30 21:54:25.592978 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:25.592628 6720 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-7v8vz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 21:54:25.592978 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:25.592688 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-7v8vz" podUID="85e8605a-0cd7-4eaf-84e2-99e19e9ebebc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 21:54:26.143872 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:26.143838 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:26.167199 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:26.166747 6720 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 21:54:26.167199 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:26.166817 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd" podUID="1817b106-b8fd-4989-977b-115eca7245ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 21:54:26.387087 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:26.387020 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:26.571457 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:26.570810 6720 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-7v8vz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 21:54:26.571457 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:26.570866 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-7v8vz" podUID="85e8605a-0cd7-4eaf-84e2-99e19e9ebebc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 21:54:27.192279 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:27.192244 6720 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 21:54:27.192885 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:27.192856 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd" podUID="1817b106-b8fd-4989-977b-115eca7245ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 21:54:27.513506 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:27.512886 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:27.573930 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:27.573894 6720 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-7v8vz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 21:54:27.574236 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:27.574192 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-7v8vz" podUID="85e8605a-0cd7-4eaf-84e2-99e19e9ebebc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 21:54:28.168798 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:28.168764 6720 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 21:54:28.169238 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:28.169207 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd" podUID="1817b106-b8fd-4989-977b-115eca7245ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 21:54:28.578867 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:28.578761 6720 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-7v8vz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 21:54:28.579533 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:28.579501 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-7v8vz" podUID="85e8605a-0cd7-4eaf-84e2-99e19e9ebebc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 21:54:28.738722 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:28.738682 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45110d7e-jj2v4"] Oct 30 21:54:28.742822 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:28.742797 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45110d7e-jj2v4" Oct 30 21:54:28.784412 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:28.780472 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:54:28.784412 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:28.780635 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 21:54:28.804948 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:28.804865 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45110d7e-jj2v4"] Oct 30 21:54:28.820860 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:28.820833 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b21b427-82ce-45a6-a795-57e67ac46259-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45110d7e-jj2v4\" (UID: \"9b21b427-82ce-45a6-a795-57e67ac46259\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45110d7e-jj2v4" Oct 30 21:54:28.821084 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:28.820997 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b21b427-82ce-45a6-a795-57e67ac46259-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45110d7e-jj2v4\" (UID: \"9b21b427-82ce-45a6-a795-57e67ac46259\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45110d7e-jj2v4" Oct 30 21:54:28.821233 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:28.821217 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xdkm\" (UniqueName: \"kubernetes.io/projected/9b21b427-82ce-45a6-a795-57e67ac46259-kube-api-access-2xdkm\") pod \"maas-api-revoke-sub-45110d7e-jj2v4\" (UID: \"9b21b427-82ce-45a6-a795-57e67ac46259\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45110d7e-jj2v4" Oct 30 21:54:28.921951 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:28.921918 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xdkm\" (UniqueName: \"kubernetes.io/projected/9b21b427-82ce-45a6-a795-57e67ac46259-kube-api-access-2xdkm\") pod \"maas-api-revoke-sub-45110d7e-jj2v4\" (UID: \"9b21b427-82ce-45a6-a795-57e67ac46259\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45110d7e-jj2v4" Oct 30 21:54:28.922126 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:28.921986 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b21b427-82ce-45a6-a795-57e67ac46259-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45110d7e-jj2v4\" (UID: \"9b21b427-82ce-45a6-a795-57e67ac46259\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45110d7e-jj2v4" Oct 30 21:54:28.922126 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:28.922032 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b21b427-82ce-45a6-a795-57e67ac46259-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45110d7e-jj2v4\" (UID: \"9b21b427-82ce-45a6-a795-57e67ac46259\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45110d7e-jj2v4" Oct 30 21:54:28.923024 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:28.923002 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b21b427-82ce-45a6-a795-57e67ac46259-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45110d7e-jj2v4\" (UID: \"9b21b427-82ce-45a6-a795-57e67ac46259\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45110d7e-jj2v4" Oct 30 21:54:28.924626 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:28.924611 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b21b427-82ce-45a6-a795-57e67ac46259-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45110d7e-jj2v4\" (UID: \"9b21b427-82ce-45a6-a795-57e67ac46259\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45110d7e-jj2v4" Oct 30 21:54:28.935713 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:28.935682 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xdkm\" (UniqueName: \"kubernetes.io/projected/9b21b427-82ce-45a6-a795-57e67ac46259-kube-api-access-2xdkm\") pod \"maas-api-revoke-sub-45110d7e-jj2v4\" (UID: \"9b21b427-82ce-45a6-a795-57e67ac46259\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45110d7e-jj2v4" Oct 30 21:54:29.104537 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:29.104502 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45110d7e-jj2v4" Oct 30 21:54:29.168406 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:29.167788 6720 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 21:54:29.168406 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:29.167856 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd" podUID="1817b106-b8fd-4989-977b-115eca7245ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 21:54:29.328422 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:29.328204 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:29.520729 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:29.508615 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-0f26c5/distinct-2-w1-0f26c5-kserve-74976f985c-fw28x"] Oct 30 21:54:29.538603 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:29.537548 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-0f26c5/distinct-2-w1-0f26c5-kserve-74976f985c-fw28x"] Oct 30 21:54:29.538603 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:29.537654 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-0f26c5/distinct-2-w1-0f26c5-kserve-74976f985c-fw28x" Oct 30 21:54:29.540731 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:54:29.539249 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b21b427_82ce_45a6_a795_57e67ac46259.slice/crio-24880fde40051ce6d84f3196a1bb674fc7b8d8b0314ba4a29ea7bd0b305c1b98 WatchSource:0}: Error finding container 24880fde40051ce6d84f3196a1bb674fc7b8d8b0314ba4a29ea7bd0b305c1b98: Status 404 returned error can't find the container with id 24880fde40051ce6d84f3196a1bb674fc7b8d8b0314ba4a29ea7bd0b305c1b98 Oct 30 21:54:29.541042 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:29.540960 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45110d7e-jj2v4"] Oct 30 21:54:29.542288 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:29.542271 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-0f26c5\"/\"distinct-2-w1-0f26c5-kserve-self-signed-certs\"" Oct 30 21:54:29.589464 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:29.588736 6720 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-7v8vz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 21:54:29.589464 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:29.588790 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-7v8vz" podUID="85e8605a-0cd7-4eaf-84e2-99e19e9ebebc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 21:54:29.590691 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:29.590482 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:29.667408 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:29.661069 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97e1e65d-1d50-4097-98ca-2944fa182807-model-cache\") pod \"distinct-2-w1-0f26c5-kserve-74976f985c-fw28x\" (UID: \"97e1e65d-1d50-4097-98ca-2944fa182807\") " pod="e2e-models-e2e-worker-w1-0f26c5/distinct-2-w1-0f26c5-kserve-74976f985c-fw28x" Oct 30 21:54:29.667408 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:29.661114 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97e1e65d-1d50-4097-98ca-2944fa182807-tls-certs\") pod \"distinct-2-w1-0f26c5-kserve-74976f985c-fw28x\" (UID: \"97e1e65d-1d50-4097-98ca-2944fa182807\") " pod="e2e-models-e2e-worker-w1-0f26c5/distinct-2-w1-0f26c5-kserve-74976f985c-fw28x" Oct 30 21:54:29.667408 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:29.661186 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97e1e65d-1d50-4097-98ca-2944fa182807-tmp-dir\") pod \"distinct-2-w1-0f26c5-kserve-74976f985c-fw28x\" (UID: \"97e1e65d-1d50-4097-98ca-2944fa182807\") " pod="e2e-models-e2e-worker-w1-0f26c5/distinct-2-w1-0f26c5-kserve-74976f985c-fw28x" Oct 30 21:54:29.667408 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:29.661235 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97e1e65d-1d50-4097-98ca-2944fa182807-home\") pod \"distinct-2-w1-0f26c5-kserve-74976f985c-fw28x\" (UID: \"97e1e65d-1d50-4097-98ca-2944fa182807\") " pod="e2e-models-e2e-worker-w1-0f26c5/distinct-2-w1-0f26c5-kserve-74976f985c-fw28x" Oct 30 21:54:29.667408 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:29.661275 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97e1e65d-1d50-4097-98ca-2944fa182807-dshm\") pod \"distinct-2-w1-0f26c5-kserve-74976f985c-fw28x\" (UID: \"97e1e65d-1d50-4097-98ca-2944fa182807\") " pod="e2e-models-e2e-worker-w1-0f26c5/distinct-2-w1-0f26c5-kserve-74976f985c-fw28x" Oct 30 21:54:29.667408 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:29.661348 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k88td\" (UniqueName: \"kubernetes.io/projected/97e1e65d-1d50-4097-98ca-2944fa182807-kube-api-access-k88td\") pod \"distinct-2-w1-0f26c5-kserve-74976f985c-fw28x\" (UID: \"97e1e65d-1d50-4097-98ca-2944fa182807\") " pod="e2e-models-e2e-worker-w1-0f26c5/distinct-2-w1-0f26c5-kserve-74976f985c-fw28x" Oct 30 21:54:29.779294 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:29.778496 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97e1e65d-1d50-4097-98ca-2944fa182807-model-cache\") pod \"distinct-2-w1-0f26c5-kserve-74976f985c-fw28x\" (UID: \"97e1e65d-1d50-4097-98ca-2944fa182807\") " pod="e2e-models-e2e-worker-w1-0f26c5/distinct-2-w1-0f26c5-kserve-74976f985c-fw28x" Oct 30 21:54:29.779294 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:29.778934 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97e1e65d-1d50-4097-98ca-2944fa182807-model-cache\") pod \"distinct-2-w1-0f26c5-kserve-74976f985c-fw28x\" (UID: \"97e1e65d-1d50-4097-98ca-2944fa182807\") " pod="e2e-models-e2e-worker-w1-0f26c5/distinct-2-w1-0f26c5-kserve-74976f985c-fw28x" Oct 30 21:54:29.788409 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:29.784356 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97e1e65d-1d50-4097-98ca-2944fa182807-tls-certs\") pod \"distinct-2-w1-0f26c5-kserve-74976f985c-fw28x\" (UID: \"97e1e65d-1d50-4097-98ca-2944fa182807\") " pod="e2e-models-e2e-worker-w1-0f26c5/distinct-2-w1-0f26c5-kserve-74976f985c-fw28x" Oct 30 21:54:29.788409 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:29.784492 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97e1e65d-1d50-4097-98ca-2944fa182807-tmp-dir\") pod \"distinct-2-w1-0f26c5-kserve-74976f985c-fw28x\" (UID: \"97e1e65d-1d50-4097-98ca-2944fa182807\") " pod="e2e-models-e2e-worker-w1-0f26c5/distinct-2-w1-0f26c5-kserve-74976f985c-fw28x" Oct 30 21:54:29.788409 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:29.784810 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97e1e65d-1d50-4097-98ca-2944fa182807-tmp-dir\") pod \"distinct-2-w1-0f26c5-kserve-74976f985c-fw28x\" (UID: \"97e1e65d-1d50-4097-98ca-2944fa182807\") " pod="e2e-models-e2e-worker-w1-0f26c5/distinct-2-w1-0f26c5-kserve-74976f985c-fw28x" Oct 30 21:54:29.792557 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:29.790339 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97e1e65d-1d50-4097-98ca-2944fa182807-home\") pod \"distinct-2-w1-0f26c5-kserve-74976f985c-fw28x\" (UID: \"97e1e65d-1d50-4097-98ca-2944fa182807\") " pod="e2e-models-e2e-worker-w1-0f26c5/distinct-2-w1-0f26c5-kserve-74976f985c-fw28x" Oct 30 21:54:29.792557 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:29.790421 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97e1e65d-1d50-4097-98ca-2944fa182807-dshm\") pod \"distinct-2-w1-0f26c5-kserve-74976f985c-fw28x\" (UID: \"97e1e65d-1d50-4097-98ca-2944fa182807\") " pod="e2e-models-e2e-worker-w1-0f26c5/distinct-2-w1-0f26c5-kserve-74976f985c-fw28x" Oct 30 21:54:29.792557 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:29.790510 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k88td\" (UniqueName: \"kubernetes.io/projected/97e1e65d-1d50-4097-98ca-2944fa182807-kube-api-access-k88td\") pod \"distinct-2-w1-0f26c5-kserve-74976f985c-fw28x\" (UID: \"97e1e65d-1d50-4097-98ca-2944fa182807\") " pod="e2e-models-e2e-worker-w1-0f26c5/distinct-2-w1-0f26c5-kserve-74976f985c-fw28x" Oct 30 21:54:29.792557 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:29.791123 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97e1e65d-1d50-4097-98ca-2944fa182807-home\") pod \"distinct-2-w1-0f26c5-kserve-74976f985c-fw28x\" (UID: \"97e1e65d-1d50-4097-98ca-2944fa182807\") " pod="e2e-models-e2e-worker-w1-0f26c5/distinct-2-w1-0f26c5-kserve-74976f985c-fw28x" Oct 30 21:54:29.809424 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:29.807155 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97e1e65d-1d50-4097-98ca-2944fa182807-dshm\") pod \"distinct-2-w1-0f26c5-kserve-74976f985c-fw28x\" (UID: \"97e1e65d-1d50-4097-98ca-2944fa182807\") " pod="e2e-models-e2e-worker-w1-0f26c5/distinct-2-w1-0f26c5-kserve-74976f985c-fw28x" Oct 30 21:54:29.829659 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:29.828691 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k88td\" (UniqueName: \"kubernetes.io/projected/97e1e65d-1d50-4097-98ca-2944fa182807-kube-api-access-k88td\") pod \"distinct-2-w1-0f26c5-kserve-74976f985c-fw28x\" (UID: \"97e1e65d-1d50-4097-98ca-2944fa182807\") " pod="e2e-models-e2e-worker-w1-0f26c5/distinct-2-w1-0f26c5-kserve-74976f985c-fw28x" Oct 30 21:54:29.829659 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:29.829277 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97e1e65d-1d50-4097-98ca-2944fa182807-tls-certs\") pod \"distinct-2-w1-0f26c5-kserve-74976f985c-fw28x\" (UID: \"97e1e65d-1d50-4097-98ca-2944fa182807\") " pod="e2e-models-e2e-worker-w1-0f26c5/distinct-2-w1-0f26c5-kserve-74976f985c-fw28x" Oct 30 21:54:29.990413 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:29.987602 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-0f26c5/distinct-2-w1-0f26c5-kserve-74976f985c-fw28x" Oct 30 21:54:30.170419 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:30.170283 6720 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 21:54:30.170419 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:30.170348 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd" podUID="1817b106-b8fd-4989-977b-115eca7245ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 21:54:30.210358 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:30.209537 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45110d7e-jj2v4" event={"ID":"9b21b427-82ce-45a6-a795-57e67ac46259","Type":"ContainerStarted","Data":"24880fde40051ce6d84f3196a1bb674fc7b8d8b0314ba4a29ea7bd0b305c1b98"} Oct 30 21:54:30.341418 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:30.341304 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-0f26c5/distinct-2-w1-0f26c5-kserve-74976f985c-fw28x"] Oct 30 21:54:30.480929 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:30.480824 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:30.571103 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:30.570748 6720 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-7v8vz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 21:54:30.571103 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:30.570811 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-7v8vz" podUID="85e8605a-0cd7-4eaf-84e2-99e19e9ebebc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 21:54:31.163724 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:31.163691 6720 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 21:54:31.164118 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:31.163756 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd" podUID="1817b106-b8fd-4989-977b-115eca7245ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 21:54:31.218675 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:31.218643 6720 generic.go:356] "Generic (PLEG): container finished" podID="9b21b427-82ce-45a6-a795-57e67ac46259" containerID="2a9f5940ba3d075223d6fefcff0e9bf0a1927aa83677401faf19d577ba1c5d29" exitCode=0 Oct 30 21:54:31.218860 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:31.218733 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45110d7e-jj2v4" event={"ID":"9b21b427-82ce-45a6-a795-57e67ac46259","Type":"ContainerDied","Data":"2a9f5940ba3d075223d6fefcff0e9bf0a1927aa83677401faf19d577ba1c5d29"} Oct 30 21:54:31.220729 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:31.220705 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-0f26c5/distinct-2-w1-0f26c5-kserve-74976f985c-fw28x" event={"ID":"97e1e65d-1d50-4097-98ca-2944fa182807","Type":"ContainerStarted","Data":"96f2583dfd9b4c597466bcd32b75582e9764458cc65ff98e4dc8c30aff0e8176"} Oct 30 21:54:31.220729 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:31.220736 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-0f26c5/distinct-2-w1-0f26c5-kserve-74976f985c-fw28x" event={"ID":"97e1e65d-1d50-4097-98ca-2944fa182807","Type":"ContainerStarted","Data":"fdc4418bc41b83d626bab237150b3c67dde8cc6e0fa5feefbfb785e1ff9b5f30"} Oct 30 21:54:31.280253 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:31.280129 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-0f26c5/distinct-2-w1-0f26c5-kserve-74976f985c-fw28x" podStartSLOduration=2.280112557 podStartE2EDuration="2.280112557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:31.2786297 +0000 UTC m=+837.449594883" watchObservedRunningTime="2026-10-30 21:54:31.280112557 +0000 UTC m=+837.451077742" Oct 30 21:54:31.301286 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:31.301251 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:31.988276 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:31.988241 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0f26c5/distinct-2-w1-0f26c5-kserve-74976f985c-fw28x" Oct 30 21:54:32.017117 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:32.017082 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:32.047403 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:32.047351 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-7v8vz" Oct 30 21:54:32.048075 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:32.048046 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-7v8vz" Oct 30 21:54:32.048765 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:32.048745 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-7v8vz" Oct 30 21:54:32.167255 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:32.167224 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd" Oct 30 21:54:32.167669 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:32.167650 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd" Oct 30 21:54:32.168316 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:32.168293 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd" Oct 30 21:54:32.386955 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:32.386928 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45110d7e-jj2v4" Oct 30 21:54:32.445034 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:32.444999 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b21b427-82ce-45a6-a795-57e67ac46259-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b21b427-82ce-45a6-a795-57e67ac46259-maas-api-openshift-ca\") pod \"9b21b427-82ce-45a6-a795-57e67ac46259\" (UID: \"9b21b427-82ce-45a6-a795-57e67ac46259\") " Oct 30 21:54:32.445222 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:32.445108 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b21b427-82ce-45a6-a795-57e67ac46259-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b21b427-82ce-45a6-a795-57e67ac46259-maas-api-xks-ca\") pod \"9b21b427-82ce-45a6-a795-57e67ac46259\" (UID: \"9b21b427-82ce-45a6-a795-57e67ac46259\") " Oct 30 21:54:32.445222 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:32.445157 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b21b427-82ce-45a6-a795-57e67ac46259-kube-api-access-2xdkm\" (UniqueName: \"kubernetes.io/projected/9b21b427-82ce-45a6-a795-57e67ac46259-kube-api-access-2xdkm\") pod \"9b21b427-82ce-45a6-a795-57e67ac46259\" (UID: \"9b21b427-82ce-45a6-a795-57e67ac46259\") " Oct 30 21:54:32.445429 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:32.445405 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b21b427-82ce-45a6-a795-57e67ac46259-maas-api-openshift-ca" pod "9b21b427-82ce-45a6-a795-57e67ac46259" (UID: "9b21b427-82ce-45a6-a795-57e67ac46259"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:54:32.445758 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:32.445733 6720 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b21b427-82ce-45a6-a795-57e67ac46259-maas-api-openshift-ca\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:54:32.447864 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:32.447836 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b21b427-82ce-45a6-a795-57e67ac46259-kube-api-access-2xdkm" pod "9b21b427-82ce-45a6-a795-57e67ac46259" (UID: "9b21b427-82ce-45a6-a795-57e67ac46259"). InnerVolumeSpecName "kube-api-access-2xdkm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:32.448027 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:32.447999 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b21b427-82ce-45a6-a795-57e67ac46259-maas-api-xks-ca" pod "9b21b427-82ce-45a6-a795-57e67ac46259" (UID: "9b21b427-82ce-45a6-a795-57e67ac46259"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:54:32.547057 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:32.546952 6720 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b21b427-82ce-45a6-a795-57e67ac46259-maas-api-xks-ca\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:54:32.547057 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:32.547013 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xdkm\" (UniqueName: \"kubernetes.io/projected/9b21b427-82ce-45a6-a795-57e67ac46259-kube-api-access-2xdkm\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:54:32.762461 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:32.762428 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:33.233749 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:33.233717 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45110d7e-jj2v4" Oct 30 21:54:33.234150 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:33.233757 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45110d7e-jj2v4" event={"ID":"9b21b427-82ce-45a6-a795-57e67ac46259","Type":"ContainerDied","Data":"24880fde40051ce6d84f3196a1bb674fc7b8d8b0314ba4a29ea7bd0b305c1b98"} Oct 30 21:54:33.234150 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:33.233807 6720 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24880fde40051ce6d84f3196a1bb674fc7b8d8b0314ba4a29ea7bd0b305c1b98" Oct 30 21:54:33.608906 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:33.608872 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:34.401006 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:34.400972 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:35.006030 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:35.005994 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:35.716232 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:35.716189 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:36.601942 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:36.601915 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:37.412745 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:37.412705 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:38.215796 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:38.215757 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:39.392029 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:39.391998 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:39.626410 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:39.625641 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:40.018240 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:40.018200 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-0f26c5/distinct-2-w1-0f26c5-kserve-74976f985c-fw28x" Oct 30 21:54:40.391423 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:40.382433 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:40.570409 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:40.570098 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:40.594688 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:54:40.594628 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 21:54:40.910411 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:40.909402 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-0f26c5/distinct-2-w1-0f26c5-kserve-74976f985c-fw28x" Oct 30 21:54:40.949412 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:40.945516 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/distinct-w1-0f26c5-kserve-7d6b79498b-lvsjw" podUID="457d2f97-63ab-41f4-b761-294d0ba3045e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:40.987422 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:40.986603 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/premium-w3-ceb36a-kserve-6576bb476f-j5kdr" podUID="d065e788-c972-4683-84ed-86a70ed4791d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:41.640416 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:41.638681 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/distinct-2-w0-1529dc-kserve-7788894c6-7mbrr" podUID="09662f19-1ef9-406c-85ef-e402505875b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:41.671412 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:41.668168 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/distinct-w4-939a16-kserve-7f88f68c56-cngtk" podUID="85316c69-7ebd-4fb6-9195-44d1fba2dfbc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:41.734702 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:41.727268 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:41.835420 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:41.834504 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/simulator-w0-1529dc-kserve-8466d8d645-l54cf" podUID="257edb08-2e86-4f49-be57-848ace5958b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:41.902836 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:41.902727 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/simulator-w4-939a16-kserve-74f845bf84-4c7b6" podUID="9bb6805f-f355-4ea8-80b7-542215c42022" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:41.942928 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:41.911467 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/distinct-2-w1-0f26c5-kserve-74976f985c-fw28x" podUID="97e1e65d-1d50-4097-98ca-2944fa182807" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:42.015915 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:42.014289 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:42.015915 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:42.014565 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/premium-w3-ceb36a-kserve-6576bb476f-j5kdr" podUID="d065e788-c972-4683-84ed-86a70ed4791d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:42.020767 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:42.020743 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ceb36a/premium-w3-ceb36a-kserve-6576bb476f-j5kdr" Oct 30 21:54:42.142058 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:42.141430 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ceb36a/premium-w3-ceb36a-kserve-6576bb476f-j5kdr" Oct 30 21:54:42.195533 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:42.194688 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0f26c5/distinct-2-w1-0f26c5-kserve-74976f985c-fw28x" Oct 30 21:54:42.609355 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:42.609318 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:43.544998 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:43.544963 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:44.377670 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:44.377638 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:45.290996 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:45.290945 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:45.939066 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:45.939033 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:46.765115 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:46.765073 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:47.440205 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:47.440137 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:48.305450 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:48.305407 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:49.030230 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:49.030201 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:50.021413 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:50.016879 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:50.268938 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:50.268900 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:50.579907 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:50.579874 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:50.725870 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:50.725834 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:51.439885 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:51.438592 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:51.672609 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:51.672549 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/distinct-w4-939a16-kserve-7f88f68c56-cngtk" podUID="85316c69-7ebd-4fb6-9195-44d1fba2dfbc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 21:54:51.726504 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:51.701004 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/distinct-2-w0-1529dc-kserve-7788894c6-7mbrr" podUID="09662f19-1ef9-406c-85ef-e402505875b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:51.853602 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:51.853542 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/simulator-w0-1529dc-kserve-8466d8d645-l54cf" podUID="257edb08-2e86-4f49-be57-848ace5958b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:51.900632 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:51.900600 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/simulator-w4-939a16-kserve-74f845bf84-4c7b6" podUID="9bb6805f-f355-4ea8-80b7-542215c42022" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:51.901934 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:51.900545 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/distinct-w1-0f26c5-kserve-7d6b79498b-lvsjw" podUID="457d2f97-63ab-41f4-b761-294d0ba3045e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:51.911873 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:51.911841 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/distinct-2-w1-0f26c5-kserve-74976f985c-fw28x" podUID="97e1e65d-1d50-4097-98ca-2944fa182807" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:52.143574 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:52.143521 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/premium-w3-ceb36a-kserve-6576bb476f-j5kdr" podUID="d065e788-c972-4683-84ed-86a70ed4791d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:52.145958 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:52.143839 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-2-w3-ceb36a-kserve-7c59695467-46gdc" podUID="f1b6e92c-2952-4bcb-ae33-91e7271342fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 30 21:54:52.462612 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:52.462503 6720 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1529dc/distinct-2-w0-1529dc-kserve-7788894c6-7mbrr" podUID="09662f19-1ef9-406c-85ef-e402505875b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:52.680218 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:52.680102 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/distinct-w4-939a16-kserve-7f88f68c56-cngtk" podUID="85316c69-7ebd-4fb6-9195-44d1fba2dfbc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:52.681424 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:52.680737 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-939a16/distinct-w4-939a16-kserve-7f88f68c56-cngtk" Oct 30 21:54:52.707026 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:52.703466 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/distinct-2-w0-1529dc-kserve-7788894c6-7mbrr" podUID="09662f19-1ef9-406c-85ef-e402505875b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:52.707026 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:52.704344 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1529dc/distinct-2-w0-1529dc-kserve-7788894c6-7mbrr" Oct 30 21:54:52.787919 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:52.785476 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-939a16/distinct-w4-939a16-kserve-7f88f68c56-cngtk" Oct 30 21:54:52.818746 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:52.818666 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:52.869776 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:52.867478 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/simulator-w0-1529dc-kserve-8466d8d645-l54cf" podUID="257edb08-2e86-4f49-be57-848ace5958b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 21:54:52.869776 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:52.868224 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1529dc/simulator-w0-1529dc-kserve-8466d8d645-l54cf" Oct 30 21:54:52.902581 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:52.902538 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/distinct-w1-0f26c5-kserve-7d6b79498b-lvsjw" podUID="457d2f97-63ab-41f4-b761-294d0ba3045e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 30 21:54:52.935123 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:52.905484 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/simulator-w4-939a16-kserve-74f845bf84-4c7b6" podUID="9bb6805f-f355-4ea8-80b7-542215c42022" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:52.935123 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:52.919338 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0f26c5/distinct-w1-0f26c5-kserve-7d6b79498b-lvsjw" Oct 30 21:54:52.935123 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:52.919425 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-939a16/simulator-w4-939a16-kserve-74f845bf84-4c7b6" Oct 30 21:54:53.033410 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:53.018586 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-79bch" podUID="7124e1e3-a3b7-4806-8a4a-609f9e2c0ae5" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.21:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:53.038835 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:53.038691 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:53.190412 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:53.161040 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-2-w3-ceb36a-kserve-7c59695467-46gdc" podUID="f1b6e92c-2952-4bcb-ae33-91e7271342fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 30 21:54:53.190412 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:53.161796 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/premium-w3-ceb36a-kserve-6576bb476f-j5kdr" podUID="d065e788-c972-4683-84ed-86a70ed4791d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 21:54:53.190412 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:53.162507 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-2-w3-ceb36a-kserve-7c59695467-46gdc" Oct 30 21:54:53.190412 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:53.162565 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ceb36a/premium-w3-ceb36a-kserve-6576bb476f-j5kdr" Oct 30 21:54:53.216420 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:53.207506 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1529dc/simulator-w0-1529dc-kserve-8466d8d645-l54cf" Oct 30 21:54:53.578993 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:53.578958 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0f26c5/distinct-w1-0f26c5-kserve-7d6b79498b-lvsjw" Oct 30 21:54:53.718679 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:53.718636 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/distinct-2-w0-1529dc-kserve-7788894c6-7mbrr" podUID="09662f19-1ef9-406c-85ef-e402505875b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 21:54:53.971887 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:53.971847 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/simulator-w4-939a16-kserve-74f845bf84-4c7b6" podUID="9bb6805f-f355-4ea8-80b7-542215c42022" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 21:54:54.137503 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:54.137364 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:54.163413 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:54.162473 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/premium-w3-ceb36a-kserve-6576bb476f-j5kdr" podUID="d065e788-c972-4683-84ed-86a70ed4791d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:54.163413 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:54.162898 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-2-w3-ceb36a-kserve-7c59695467-46gdc" podUID="f1b6e92c-2952-4bcb-ae33-91e7271342fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 30 21:54:54.205602 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:54.204824 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/simulator-w0-1529dc-kserve-8466d8d645-l54cf" podUID="257edb08-2e86-4f49-be57-848ace5958b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:54.258525 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:54.256299 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-939a16/simulator-w4-939a16-kserve-74f845bf84-4c7b6" Oct 30 21:54:54.536096 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:54.531408 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:54.808351 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:54.808261 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/distinct-2-w0-1529dc-kserve-7788894c6-7mbrr" podUID="09662f19-1ef9-406c-85ef-e402505875b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:55.134838 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:55.134806 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1529dc/distinct-2-w0-1529dc-kserve-7788894c6-7mbrr" Oct 30 21:54:55.169518 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:55.169473 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/premium-w3-ceb36a-kserve-6576bb476f-j5kdr" podUID="d065e788-c972-4683-84ed-86a70ed4791d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:55.248228 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:55.210075 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-2-w3-ceb36a-kserve-7c59695467-46gdc" podUID="f1b6e92c-2952-4bcb-ae33-91e7271342fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:55.546815 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:55.546677 6720 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-2-w3-ceb36a-kserve-7c59695467-46gdc" podUID="f1b6e92c-2952-4bcb-ae33-91e7271342fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:55.791639 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:55.791593 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/distinct-w4-939a16-kserve-7f88f68c56-cngtk" podUID="85316c69-7ebd-4fb6-9195-44d1fba2dfbc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 21:54:55.922530 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:55.922487 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/distinct-2-w1-0f26c5-kserve-74976f985c-fw28x" podUID="97e1e65d-1d50-4097-98ca-2944fa182807" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:56.025978 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:56.023536 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-2tv9m"] Oct 30 21:54:56.025978 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:56.024260 6720 cpu_manager.go:414] "containerMap: removing container" podUID="9b21b427-82ce-45a6-a795-57e67ac46259" containerName="revoke-keys" Oct 30 21:54:56.025978 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:56.024274 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b21b427-82ce-45a6-a795-57e67ac46259" containerName="revoke-keys" Oct 30 21:54:56.025978 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:56.024455 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b21b427-82ce-45a6-a795-57e67ac46259" containerName="revoke-keys" Oct 30 21:54:56.126726 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:56.126693 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-2tv9m" Oct 30 21:54:56.145501 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:56.133624 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-2tv9m"] Oct 30 21:54:56.160835 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:56.159983 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-0f26c5-openshift-default-dockercfg-tmpf5\"" Oct 30 21:54:56.258911 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:56.258154 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/premium-w3-ceb36a-kserve-6576bb476f-j5kdr" podUID="d065e788-c972-4683-84ed-86a70ed4791d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:56.260656 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:56.260278 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/simulator-w0-1529dc-kserve-8466d8d645-l54cf" podUID="257edb08-2e86-4f49-be57-848ace5958b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:56.302435 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:56.300656 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-2-w3-ceb36a-kserve-7c59695467-46gdc" podUID="f1b6e92c-2952-4bcb-ae33-91e7271342fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:56.302435 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:56.301174 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/simulator-w4-939a16-kserve-74f845bf84-4c7b6" podUID="9bb6805f-f355-4ea8-80b7-542215c42022" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 21:54:56.373201 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:56.373167 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bd82fe0-1de3-40c8-b825-51eff2ae3386-credential-socket\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-2tv9m\" (UID: \"0bd82fe0-1de3-40c8-b825-51eff2ae3386\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-2tv9m" Oct 30 21:54:56.377601 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:56.377552 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxnsd\" (UniqueName: \"kubernetes.io/projected/0bd82fe0-1de3-40c8-b825-51eff2ae3386-kube-api-access-zxnsd\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-2tv9m\" (UID: \"0bd82fe0-1de3-40c8-b825-51eff2ae3386\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-2tv9m" Oct 30 21:54:56.377874 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:56.377829 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bd82fe0-1de3-40c8-b825-51eff2ae3386-istiod-ca-cert\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-2tv9m\" (UID: \"0bd82fe0-1de3-40c8-b825-51eff2ae3386\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-2tv9m" Oct 30 21:54:56.378198 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:56.378178 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bd82fe0-1de3-40c8-b825-51eff2ae3386-workload-socket\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-2tv9m\" (UID: \"0bd82fe0-1de3-40c8-b825-51eff2ae3386\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-2tv9m" Oct 30 21:54:56.378492 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:56.378472 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bd82fe0-1de3-40c8-b825-51eff2ae3386-istio-data\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-2tv9m\" (UID: \"0bd82fe0-1de3-40c8-b825-51eff2ae3386\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-2tv9m" Oct 30 21:54:56.378685 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:56.378644 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bd82fe0-1de3-40c8-b825-51eff2ae3386-istio-envoy\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-2tv9m\" (UID: \"0bd82fe0-1de3-40c8-b825-51eff2ae3386\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-2tv9m" Oct 30 21:54:56.378997 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:56.378975 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bd82fe0-1de3-40c8-b825-51eff2ae3386-workload-certs\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-2tv9m\" (UID: \"0bd82fe0-1de3-40c8-b825-51eff2ae3386\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-2tv9m" Oct 30 21:54:56.379197 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:56.379164 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bd82fe0-1de3-40c8-b825-51eff2ae3386-istio-token\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-2tv9m\" (UID: \"0bd82fe0-1de3-40c8-b825-51eff2ae3386\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-2tv9m" Oct 30 21:54:56.379455 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:56.379435 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bd82fe0-1de3-40c8-b825-51eff2ae3386-istio-podinfo\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-2tv9m\" (UID: \"0bd82fe0-1de3-40c8-b825-51eff2ae3386\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-2tv9m" Oct 30 21:54:56.393225 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:56.393204 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ceb36a/premium-w3-ceb36a-kserve-6576bb476f-j5kdr" Oct 30 21:54:56.491720 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:56.491669 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bd82fe0-1de3-40c8-b825-51eff2ae3386-workload-socket\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-2tv9m\" (UID: \"0bd82fe0-1de3-40c8-b825-51eff2ae3386\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-2tv9m" Oct 30 21:54:56.492223 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:56.492198 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bd82fe0-1de3-40c8-b825-51eff2ae3386-istio-data\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-2tv9m\" (UID: \"0bd82fe0-1de3-40c8-b825-51eff2ae3386\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-2tv9m" Oct 30 21:54:56.492473 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:56.492429 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bd82fe0-1de3-40c8-b825-51eff2ae3386-istio-envoy\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-2tv9m\" (UID: \"0bd82fe0-1de3-40c8-b825-51eff2ae3386\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-2tv9m" Oct 30 21:54:56.494832 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:56.494805 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bd82fe0-1de3-40c8-b825-51eff2ae3386-workload-socket\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-2tv9m\" (UID: \"0bd82fe0-1de3-40c8-b825-51eff2ae3386\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-2tv9m" Oct 30 21:54:56.495237 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:56.495216 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bd82fe0-1de3-40c8-b825-51eff2ae3386-istio-data\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-2tv9m\" (UID: \"0bd82fe0-1de3-40c8-b825-51eff2ae3386\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-2tv9m" Oct 30 21:54:56.497496 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:56.497477 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bd82fe0-1de3-40c8-b825-51eff2ae3386-workload-certs\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-2tv9m\" (UID: \"0bd82fe0-1de3-40c8-b825-51eff2ae3386\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-2tv9m" Oct 30 21:54:56.497689 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:56.497672 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bd82fe0-1de3-40c8-b825-51eff2ae3386-istio-token\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-2tv9m\" (UID: \"0bd82fe0-1de3-40c8-b825-51eff2ae3386\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-2tv9m" Oct 30 21:54:56.497835 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:56.497820 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bd82fe0-1de3-40c8-b825-51eff2ae3386-istio-podinfo\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-2tv9m\" (UID: \"0bd82fe0-1de3-40c8-b825-51eff2ae3386\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-2tv9m" Oct 30 21:54:56.497992 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:56.497979 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bd82fe0-1de3-40c8-b825-51eff2ae3386-credential-socket\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-2tv9m\" (UID: \"0bd82fe0-1de3-40c8-b825-51eff2ae3386\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-2tv9m" Oct 30 21:54:56.499176 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:56.499154 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxnsd\" (UniqueName: \"kubernetes.io/projected/0bd82fe0-1de3-40c8-b825-51eff2ae3386-kube-api-access-zxnsd\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-2tv9m\" (UID: \"0bd82fe0-1de3-40c8-b825-51eff2ae3386\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-2tv9m" Oct 30 21:54:56.499325 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:56.499309 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bd82fe0-1de3-40c8-b825-51eff2ae3386-istiod-ca-cert\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-2tv9m\" (UID: \"0bd82fe0-1de3-40c8-b825-51eff2ae3386\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-2tv9m" Oct 30 21:54:56.528461 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:56.500149 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bd82fe0-1de3-40c8-b825-51eff2ae3386-istiod-ca-cert\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-2tv9m\" (UID: \"0bd82fe0-1de3-40c8-b825-51eff2ae3386\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-2tv9m" Oct 30 21:54:56.528461 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:56.498802 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bd82fe0-1de3-40c8-b825-51eff2ae3386-credential-socket\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-2tv9m\" (UID: \"0bd82fe0-1de3-40c8-b825-51eff2ae3386\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-2tv9m" Oct 30 21:54:56.528461 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:56.509936 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bd82fe0-1de3-40c8-b825-51eff2ae3386-workload-certs\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-2tv9m\" (UID: \"0bd82fe0-1de3-40c8-b825-51eff2ae3386\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-2tv9m" Oct 30 21:54:56.623840 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:56.596373 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/distinct-w1-0f26c5-kserve-7d6b79498b-lvsjw" podUID="457d2f97-63ab-41f4-b761-294d0ba3045e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 30 21:54:56.721876 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:56.721840 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:56.741246 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:56.737845 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bd82fe0-1de3-40c8-b825-51eff2ae3386-istio-envoy\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-2tv9m\" (UID: \"0bd82fe0-1de3-40c8-b825-51eff2ae3386\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-2tv9m" Oct 30 21:54:56.741246 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:56.738525 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bd82fe0-1de3-40c8-b825-51eff2ae3386-istio-token\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-2tv9m\" (UID: \"0bd82fe0-1de3-40c8-b825-51eff2ae3386\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-2tv9m" Oct 30 21:54:56.761680 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:56.759565 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxnsd\" (UniqueName: \"kubernetes.io/projected/0bd82fe0-1de3-40c8-b825-51eff2ae3386-kube-api-access-zxnsd\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-2tv9m\" (UID: \"0bd82fe0-1de3-40c8-b825-51eff2ae3386\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-2tv9m" Oct 30 21:54:56.763348 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:56.762341 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bd82fe0-1de3-40c8-b825-51eff2ae3386-istio-podinfo\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-2tv9m\" (UID: \"0bd82fe0-1de3-40c8-b825-51eff2ae3386\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-2tv9m" Oct 30 21:54:56.788662 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:56.787963 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-2tv9m" Oct 30 21:54:56.827594 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:56.827554 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/distinct-2-w0-1529dc-kserve-7788894c6-7mbrr" podUID="09662f19-1ef9-406c-85ef-e402505875b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:54:56.938331 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:56.938295 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-2-w3-ceb36a-kserve-7c59695467-46gdc" Oct 30 21:54:56.945988 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:56.945939 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:57.307822 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:57.302620 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-2tv9m"] Oct 30 21:54:57.317876 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:54:57.311934 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bd82fe0_1de3_40c8_b825_51eff2ae3386.slice/crio-e00099c03bc7cae9fd4819887f91fbb82e71daf43a7b3ec0d0076abba3fddee3 WatchSource:0}: Error finding container e00099c03bc7cae9fd4819887f91fbb82e71daf43a7b3ec0d0076abba3fddee3: Status 404 returned error can't find the container with id e00099c03bc7cae9fd4819887f91fbb82e71daf43a7b3ec0d0076abba3fddee3 Oct 30 21:54:57.344953 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:57.333080 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:54:57.344953 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:57.333198 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:54:57.344953 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:57.333264 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:54:58.121457 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:58.121422 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:58.181081 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:58.181042 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-2tv9m" event={"ID":"0bd82fe0-1de3-40c8-b825-51eff2ae3386","Type":"ContainerStarted","Data":"baba776f8ac85fccfd4545d6dbdbe219f96bee72ac4a9b7dd9da08595218b497"} Oct 30 21:54:58.181271 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:58.181135 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-2tv9m" event={"ID":"0bd82fe0-1de3-40c8-b825-51eff2ae3386","Type":"ContainerStarted","Data":"e00099c03bc7cae9fd4819887f91fbb82e71daf43a7b3ec0d0076abba3fddee3"} Oct 30 21:54:58.789454 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:58.789419 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-2tv9m" Oct 30 21:54:58.791340 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:58.791314 6720 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-2tv9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 21:54:58.791511 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:58.791362 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-2tv9m" podUID="0bd82fe0-1de3-40c8-b825-51eff2ae3386" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 21:54:59.424408 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:59.422823 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:54:59.790323 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:59.790235 6720 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-2tv9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 21:54:59.790525 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:59.790312 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-2tv9m" podUID="0bd82fe0-1de3-40c8-b825-51eff2ae3386" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 21:54:59.918305 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:54:59.918274 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:00.162411 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:00.162006 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:00.699960 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:00.699927 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:00.789955 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:00.789924 6720 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-2tv9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 21:55:00.790136 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:00.789988 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-2tv9m" podUID="0bd82fe0-1de3-40c8-b825-51eff2ae3386" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 21:55:01.271801 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:01.270455 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:01.496260 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:01.493788 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:01.789833 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:01.789802 6720 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-2tv9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 21:55:01.790226 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:01.789859 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-2tv9m" podUID="0bd82fe0-1de3-40c8-b825-51eff2ae3386" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 21:55:02.084457 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:02.084351 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:02.790344 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:02.790311 6720 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-2tv9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 21:55:02.790758 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:02.790363 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-2tv9m" podUID="0bd82fe0-1de3-40c8-b825-51eff2ae3386" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 21:55:03.075827 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:03.075739 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:03.790246 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:03.790214 6720 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-2tv9m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 21:55:03.790487 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:03.790270 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-2tv9m" podUID="0bd82fe0-1de3-40c8-b825-51eff2ae3386" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 21:55:04.043564 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:04.043485 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:04.198267 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:04.198234 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:04.860451 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:04.860378 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:05.278929 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:05.278833 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-2tv9m" Oct 30 21:55:05.279270 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:05.279206 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-2tv9m" Oct 30 21:55:05.280103 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:05.280084 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-2tv9m" Oct 30 21:55:05.305935 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:05.305885 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-2tv9m" podStartSLOduration=10.305871812 podStartE2EDuration="10.305871812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:58.225672009 +0000 UTC m=+864.396637192" watchObservedRunningTime="2026-10-30 21:55:05.305871812 +0000 UTC m=+871.476836994" Oct 30 21:55:05.527023 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:05.526994 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-689a2563-7b6b9dcb5f-mnxgq"] Oct 30 21:55:05.532417 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:05.532302 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-689a2563-7b6b9dcb5f-mnxgq" Oct 30 21:55:05.538287 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:05.538243 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-689a2563-serving-cert\"" Oct 30 21:55:05.541467 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:05.541443 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-689a2563-7b6b9dcb5f-mnxgq"] Oct 30 21:55:05.601937 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:05.601906 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca0e6c0a-6b9d-4b0a-b10a-69eaad73dc33-maas-api-tls\") pod \"maas-api-e2e-mt-689a2563-7b6b9dcb5f-mnxgq\" (UID: \"ca0e6c0a-6b9d-4b0a-b10a-69eaad73dc33\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-689a2563-7b6b9dcb5f-mnxgq" Oct 30 21:55:05.601937 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:05.601936 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca0e6c0a-6b9d-4b0a-b10a-69eaad73dc33-metrics-certs\") pod \"maas-api-e2e-mt-689a2563-7b6b9dcb5f-mnxgq\" (UID: \"ca0e6c0a-6b9d-4b0a-b10a-69eaad73dc33\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-689a2563-7b6b9dcb5f-mnxgq" Oct 30 21:55:05.602134 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:05.602030 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zlsr\" (UniqueName: \"kubernetes.io/projected/ca0e6c0a-6b9d-4b0a-b10a-69eaad73dc33-kube-api-access-9zlsr\") pod \"maas-api-e2e-mt-689a2563-7b6b9dcb5f-mnxgq\" (UID: \"ca0e6c0a-6b9d-4b0a-b10a-69eaad73dc33\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-689a2563-7b6b9dcb5f-mnxgq" Oct 30 21:55:05.602134 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:05.602112 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca0e6c0a-6b9d-4b0a-b10a-69eaad73dc33-openshift-service-ca\") pod \"maas-api-e2e-mt-689a2563-7b6b9dcb5f-mnxgq\" (UID: \"ca0e6c0a-6b9d-4b0a-b10a-69eaad73dc33\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-689a2563-7b6b9dcb5f-mnxgq" Oct 30 21:55:05.624680 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:05.624653 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:05.703478 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:05.703443 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca0e6c0a-6b9d-4b0a-b10a-69eaad73dc33-maas-api-tls\") pod \"maas-api-e2e-mt-689a2563-7b6b9dcb5f-mnxgq\" (UID: \"ca0e6c0a-6b9d-4b0a-b10a-69eaad73dc33\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-689a2563-7b6b9dcb5f-mnxgq" Oct 30 21:55:05.703478 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:05.703479 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca0e6c0a-6b9d-4b0a-b10a-69eaad73dc33-metrics-certs\") pod \"maas-api-e2e-mt-689a2563-7b6b9dcb5f-mnxgq\" (UID: \"ca0e6c0a-6b9d-4b0a-b10a-69eaad73dc33\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-689a2563-7b6b9dcb5f-mnxgq" Oct 30 21:55:05.703686 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:05.703574 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zlsr\" (UniqueName: \"kubernetes.io/projected/ca0e6c0a-6b9d-4b0a-b10a-69eaad73dc33-kube-api-access-9zlsr\") pod \"maas-api-e2e-mt-689a2563-7b6b9dcb5f-mnxgq\" (UID: \"ca0e6c0a-6b9d-4b0a-b10a-69eaad73dc33\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-689a2563-7b6b9dcb5f-mnxgq" Oct 30 21:55:05.703686 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:05.703633 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca0e6c0a-6b9d-4b0a-b10a-69eaad73dc33-openshift-service-ca\") pod \"maas-api-e2e-mt-689a2563-7b6b9dcb5f-mnxgq\" (UID: \"ca0e6c0a-6b9d-4b0a-b10a-69eaad73dc33\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-689a2563-7b6b9dcb5f-mnxgq" Oct 30 21:55:05.704308 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:05.704292 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca0e6c0a-6b9d-4b0a-b10a-69eaad73dc33-openshift-service-ca\") pod \"maas-api-e2e-mt-689a2563-7b6b9dcb5f-mnxgq\" (UID: \"ca0e6c0a-6b9d-4b0a-b10a-69eaad73dc33\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-689a2563-7b6b9dcb5f-mnxgq" Oct 30 21:55:05.706500 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:05.706475 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca0e6c0a-6b9d-4b0a-b10a-69eaad73dc33-metrics-certs\") pod \"maas-api-e2e-mt-689a2563-7b6b9dcb5f-mnxgq\" (UID: \"ca0e6c0a-6b9d-4b0a-b10a-69eaad73dc33\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-689a2563-7b6b9dcb5f-mnxgq" Oct 30 21:55:05.706721 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:05.706514 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca0e6c0a-6b9d-4b0a-b10a-69eaad73dc33-maas-api-tls\") pod \"maas-api-e2e-mt-689a2563-7b6b9dcb5f-mnxgq\" (UID: \"ca0e6c0a-6b9d-4b0a-b10a-69eaad73dc33\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-689a2563-7b6b9dcb5f-mnxgq" Oct 30 21:55:05.718080 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:05.718046 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zlsr\" (UniqueName: \"kubernetes.io/projected/ca0e6c0a-6b9d-4b0a-b10a-69eaad73dc33-kube-api-access-9zlsr\") pod \"maas-api-e2e-mt-689a2563-7b6b9dcb5f-mnxgq\" (UID: \"ca0e6c0a-6b9d-4b0a-b10a-69eaad73dc33\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-689a2563-7b6b9dcb5f-mnxgq" Oct 30 21:55:05.775128 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:05.775089 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:05.851285 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:05.851203 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-689a2563-7b6b9dcb5f-mnxgq" Oct 30 21:55:06.135705 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:06.135672 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-689a2563-7b6b9dcb5f-mnxgq"] Oct 30 21:55:06.145427 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:55:06.143651 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca0e6c0a_6b9d_4b0a_b10a_69eaad73dc33.slice/crio-87537dd0336458ce1b93a893f2ee21a0ede85548d98525ff0e527e5ce3b4a577 WatchSource:0}: Error finding container 87537dd0336458ce1b93a893f2ee21a0ede85548d98525ff0e527e5ce3b4a577: Status 404 returned error can't find the container with id 87537dd0336458ce1b93a893f2ee21a0ede85548d98525ff0e527e5ce3b4a577 Oct 30 21:55:06.145427 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:06.145245 6720 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:55:06.251408 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:06.249012 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-689a2563-7b6b9dcb5f-mnxgq" event={"ID":"ca0e6c0a-6b9d-4b0a-b10a-69eaad73dc33","Type":"ContainerStarted","Data":"87537dd0336458ce1b93a893f2ee21a0ede85548d98525ff0e527e5ce3b4a577"} Oct 30 21:55:06.646151 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:06.646118 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:07.257428 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:07.256318 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-689a2563-7b6b9dcb5f-mnxgq" event={"ID":"ca0e6c0a-6b9d-4b0a-b10a-69eaad73dc33","Type":"ContainerStarted","Data":"0cc0952e9cdf2848a1e571201ee38967a93d2fb000828b971aae22ec000de204"} Oct 30 21:55:07.257428 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:07.256603 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-689a2563-7b6b9dcb5f-mnxgq" Oct 30 21:55:07.278747 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:07.278702 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-689a2563-7b6b9dcb5f-mnxgq" podStartSLOduration=1.772015799 podStartE2EDuration="2.278690901s" totalImagesPullingTime="506.675102ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:07.276822005 +0000 UTC m=+873.447787187" watchObservedRunningTime="2026-10-30 21:55:07.278690901 +0000 UTC m=+873.449656083" Oct 30 21:55:07.442300 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:07.442264 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:08.144160 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:08.144129 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:08.843993 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:08.843957 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:09.536947 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:09.536895 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:10.232946 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:10.232916 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:10.404858 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:55:10.404828 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 21:55:11.052028 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:11.052000 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:11.707739 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:11.707705 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:11.862439 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:11.862377 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:12.429693 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:12.429666 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:13.308598 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:13.308568 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-689a2563-7b6b9dcb5f-mnxgq" Oct 30 21:55:13.883412 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:13.880289 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:14.219763 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:14.218490 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/simulator-w0-1529dc-kserve-8466d8d645-l54cf" podUID="257edb08-2e86-4f49-be57-848ace5958b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:55:14.607554 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:14.597667 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/distinct-w1-0f26c5-kserve-7d6b79498b-lvsjw" podUID="457d2f97-63ab-41f4-b761-294d0ba3045e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:55:14.829258 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:14.829219 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/distinct-2-w0-1529dc-kserve-7788894c6-7mbrr" podUID="09662f19-1ef9-406c-85ef-e402505875b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:55:15.171531 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:15.171477 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-2-w3-ceb36a-kserve-7c59695467-46gdc" podUID="f1b6e92c-2952-4bcb-ae33-91e7271342fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 30 21:55:15.269447 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:15.266485 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/premium-w3-ceb36a-kserve-6576bb476f-j5kdr" podUID="d065e788-c972-4683-84ed-86a70ed4791d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:55:15.302418 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:15.293493 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/simulator-w0-1529dc-kserve-8466d8d645-l54cf" podUID="257edb08-2e86-4f49-be57-848ace5958b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:55:15.341435 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:15.303215 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/simulator-w4-939a16-kserve-74f845bf84-4c7b6" podUID="9bb6805f-f355-4ea8-80b7-542215c42022" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:55:15.366773 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:15.340030 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1529dc/simulator-w0-1529dc-kserve-8466d8d645-l54cf" Oct 30 21:55:15.587962 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:15.587862 6720 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-2-w3-ceb36a-kserve-7c59695467-46gdc" podUID="f1b6e92c-2952-4bcb-ae33-91e7271342fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:55:15.610978 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:15.608804 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/distinct-w1-0f26c5-kserve-7d6b79498b-lvsjw" podUID="457d2f97-63ab-41f4-b761-294d0ba3045e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:55:15.610978 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:15.609501 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0f26c5/distinct-w1-0f26c5-kserve-7d6b79498b-lvsjw" Oct 30 21:55:15.830547 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:15.830504 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/distinct-2-w0-1529dc-kserve-7788894c6-7mbrr" podUID="09662f19-1ef9-406c-85ef-e402505875b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:55:15.831541 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:15.831519 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1529dc/distinct-2-w0-1529dc-kserve-7788894c6-7mbrr" Oct 30 21:55:15.914428 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:15.914365 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/distinct-2-w1-0f26c5-kserve-74976f985c-fw28x" podUID="97e1e65d-1d50-4097-98ca-2944fa182807" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:55:16.177570 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:16.177470 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-2-w3-ceb36a-kserve-7c59695467-46gdc" podUID="f1b6e92c-2952-4bcb-ae33-91e7271342fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:55:16.303408 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:16.295091 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/premium-w3-ceb36a-kserve-6576bb476f-j5kdr" podUID="d065e788-c972-4683-84ed-86a70ed4791d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:55:16.303408 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:16.295898 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-2-w3-ceb36a-kserve-7c59695467-46gdc" Oct 30 21:55:16.303408 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:16.295971 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0f26c5/distinct-w1-0f26c5-kserve-7d6b79498b-lvsjw" Oct 30 21:55:16.303408 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:16.296007 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1529dc/simulator-w0-1529dc-kserve-8466d8d645-l54cf" Oct 30 21:55:16.303408 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:16.296063 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ceb36a/premium-w3-ceb36a-kserve-6576bb476f-j5kdr" Oct 30 21:55:16.303408 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:16.302494 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1529dc/distinct-2-w0-1529dc-kserve-7788894c6-7mbrr" Oct 30 21:55:16.323414 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:16.306211 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:16.369551 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:16.369480 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/simulator-w4-939a16-kserve-74f845bf84-4c7b6" podUID="9bb6805f-f355-4ea8-80b7-542215c42022" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 21:55:16.372214 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:16.371622 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-939a16/simulator-w4-939a16-kserve-74f845bf84-4c7b6" Oct 30 21:55:16.420637 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:16.420570 6720 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ceb36a/premium-w3-ceb36a-kserve-6576bb476f-j5kdr" podUID="d065e788-c972-4683-84ed-86a70ed4791d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:55:16.437277 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:16.431532 6720 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:55:16.437277 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:16.431586 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269" podUID="01205782-af7b-4eda-894d-3850de540135" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:55:16.542249 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:16.540168 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ceb36a/premium-w3-ceb36a-kserve-6576bb476f-j5kdr" Oct 30 21:55:16.588481 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:16.588455 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-939a16/simulator-w4-939a16-kserve-74f845bf84-4c7b6" Oct 30 21:55:16.623914 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:16.623885 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-2-w3-ceb36a-kserve-7c59695467-46gdc" Oct 30 21:55:17.587425 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:17.587377 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:18.420703 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:18.420670 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:18.818928 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:18.818846 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:19.369805 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:19.369777 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:20.431910 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:20.431876 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:21.238431 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:21.238366 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:22.017028 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:22.016997 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:22.725123 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:22.725088 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:22.996323 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:22.996236 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4fhws"] Oct 30 21:55:23.000841 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:23.000792 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4fhws" Oct 30 21:55:23.016810 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:23.016786 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4fhws"] Oct 30 21:55:23.087401 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:23.087363 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e578cfa-7359-4c08-a8d0-56989d62227d-istiod-ca-cert\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4fhws\" (UID: \"3e578cfa-7359-4c08-a8d0-56989d62227d\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4fhws" Oct 30 21:55:23.087844 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:23.087534 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e578cfa-7359-4c08-a8d0-56989d62227d-credential-socket\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4fhws\" (UID: \"3e578cfa-7359-4c08-a8d0-56989d62227d\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4fhws" Oct 30 21:55:23.087844 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:23.087595 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfkn2\" (UniqueName: \"kubernetes.io/projected/3e578cfa-7359-4c08-a8d0-56989d62227d-kube-api-access-hfkn2\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4fhws\" (UID: \"3e578cfa-7359-4c08-a8d0-56989d62227d\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4fhws" Oct 30 21:55:23.087844 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:23.087681 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e578cfa-7359-4c08-a8d0-56989d62227d-istio-data\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4fhws\" (UID: \"3e578cfa-7359-4c08-a8d0-56989d62227d\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4fhws" Oct 30 21:55:23.087844 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:23.087717 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e578cfa-7359-4c08-a8d0-56989d62227d-istio-podinfo\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4fhws\" (UID: \"3e578cfa-7359-4c08-a8d0-56989d62227d\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4fhws" Oct 30 21:55:23.088068 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:23.087862 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e578cfa-7359-4c08-a8d0-56989d62227d-workload-certs\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4fhws\" (UID: \"3e578cfa-7359-4c08-a8d0-56989d62227d\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4fhws" Oct 30 21:55:23.088068 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:23.087924 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e578cfa-7359-4c08-a8d0-56989d62227d-workload-socket\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4fhws\" (UID: \"3e578cfa-7359-4c08-a8d0-56989d62227d\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4fhws" Oct 30 21:55:23.088068 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:23.088002 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e578cfa-7359-4c08-a8d0-56989d62227d-istio-envoy\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4fhws\" (UID: \"3e578cfa-7359-4c08-a8d0-56989d62227d\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4fhws" Oct 30 21:55:23.088201 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:23.088073 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e578cfa-7359-4c08-a8d0-56989d62227d-istio-token\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4fhws\" (UID: \"3e578cfa-7359-4c08-a8d0-56989d62227d\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4fhws" Oct 30 21:55:23.189767 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:23.189729 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e578cfa-7359-4c08-a8d0-56989d62227d-workload-certs\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4fhws\" (UID: \"3e578cfa-7359-4c08-a8d0-56989d62227d\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4fhws" Oct 30 21:55:23.189944 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:23.189794 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e578cfa-7359-4c08-a8d0-56989d62227d-workload-socket\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4fhws\" (UID: \"3e578cfa-7359-4c08-a8d0-56989d62227d\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4fhws" Oct 30 21:55:23.189944 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:23.189829 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e578cfa-7359-4c08-a8d0-56989d62227d-istio-envoy\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4fhws\" (UID: \"3e578cfa-7359-4c08-a8d0-56989d62227d\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4fhws" Oct 30 21:55:23.189944 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:23.189869 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e578cfa-7359-4c08-a8d0-56989d62227d-istio-token\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4fhws\" (UID: \"3e578cfa-7359-4c08-a8d0-56989d62227d\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4fhws" Oct 30 21:55:23.190105 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:23.189968 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e578cfa-7359-4c08-a8d0-56989d62227d-istiod-ca-cert\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4fhws\" (UID: \"3e578cfa-7359-4c08-a8d0-56989d62227d\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4fhws" Oct 30 21:55:23.190105 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:23.190046 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e578cfa-7359-4c08-a8d0-56989d62227d-credential-socket\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4fhws\" (UID: \"3e578cfa-7359-4c08-a8d0-56989d62227d\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4fhws" Oct 30 21:55:23.190211 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:23.190111 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfkn2\" (UniqueName: \"kubernetes.io/projected/3e578cfa-7359-4c08-a8d0-56989d62227d-kube-api-access-hfkn2\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4fhws\" (UID: \"3e578cfa-7359-4c08-a8d0-56989d62227d\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4fhws" Oct 30 21:55:23.190211 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:23.190154 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e578cfa-7359-4c08-a8d0-56989d62227d-workload-certs\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4fhws\" (UID: \"3e578cfa-7359-4c08-a8d0-56989d62227d\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4fhws" Oct 30 21:55:23.190211 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:23.190156 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e578cfa-7359-4c08-a8d0-56989d62227d-istio-data\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4fhws\" (UID: \"3e578cfa-7359-4c08-a8d0-56989d62227d\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4fhws" Oct 30 21:55:23.190372 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:23.190213 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e578cfa-7359-4c08-a8d0-56989d62227d-istio-podinfo\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4fhws\" (UID: \"3e578cfa-7359-4c08-a8d0-56989d62227d\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4fhws" Oct 30 21:55:23.190372 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:23.190214 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e578cfa-7359-4c08-a8d0-56989d62227d-workload-socket\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4fhws\" (UID: \"3e578cfa-7359-4c08-a8d0-56989d62227d\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4fhws" Oct 30 21:55:23.190509 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:23.190483 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e578cfa-7359-4c08-a8d0-56989d62227d-istio-data\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4fhws\" (UID: \"3e578cfa-7359-4c08-a8d0-56989d62227d\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4fhws" Oct 30 21:55:23.190555 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:23.190521 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e578cfa-7359-4c08-a8d0-56989d62227d-credential-socket\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4fhws\" (UID: \"3e578cfa-7359-4c08-a8d0-56989d62227d\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4fhws" Oct 30 21:55:23.190717 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:23.190693 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e578cfa-7359-4c08-a8d0-56989d62227d-istiod-ca-cert\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4fhws\" (UID: \"3e578cfa-7359-4c08-a8d0-56989d62227d\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4fhws" Oct 30 21:55:23.192697 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:23.192670 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e578cfa-7359-4c08-a8d0-56989d62227d-istio-envoy\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4fhws\" (UID: \"3e578cfa-7359-4c08-a8d0-56989d62227d\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4fhws" Oct 30 21:55:23.192794 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:23.192729 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e578cfa-7359-4c08-a8d0-56989d62227d-istio-podinfo\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4fhws\" (UID: \"3e578cfa-7359-4c08-a8d0-56989d62227d\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4fhws" Oct 30 21:55:23.202723 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:23.202695 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e578cfa-7359-4c08-a8d0-56989d62227d-istio-token\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4fhws\" (UID: \"3e578cfa-7359-4c08-a8d0-56989d62227d\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4fhws" Oct 30 21:55:23.203448 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:23.203403 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfkn2\" (UniqueName: \"kubernetes.io/projected/3e578cfa-7359-4c08-a8d0-56989d62227d-kube-api-access-hfkn2\") pod \"e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4fhws\" (UID: \"3e578cfa-7359-4c08-a8d0-56989d62227d\") " pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4fhws" Oct 30 21:55:23.319210 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:23.319128 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4fhws" Oct 30 21:55:23.470987 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:23.470948 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:23.621095 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:23.621070 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4fhws"] Oct 30 21:55:23.622398 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:55:23.622358 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e578cfa_7359_4c08_a8d0_56989d62227d.slice/crio-0e86e59d82f19205fb9d83a1588a39aa78a7f1ba966b286e89c99b3d04d84ee2 WatchSource:0}: Error finding container 0e86e59d82f19205fb9d83a1588a39aa78a7f1ba966b286e89c99b3d04d84ee2: Status 404 returned error can't find the container with id 0e86e59d82f19205fb9d83a1588a39aa78a7f1ba966b286e89c99b3d04d84ee2 Oct 30 21:55:23.629823 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:23.629788 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:55:23.629892 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:23.629862 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:55:23.629939 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:23.629897 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:55:23.786153 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:23.786111 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4fhws" event={"ID":"3e578cfa-7359-4c08-a8d0-56989d62227d","Type":"ContainerStarted","Data":"3fd6d374f6e0e9ab049c49320a0c11e1f964feb2dad0c1c081b6e4c3c69208f2"} Oct 30 21:55:23.786314 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:23.786186 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4fhws" event={"ID":"3e578cfa-7359-4c08-a8d0-56989d62227d","Type":"ContainerStarted","Data":"0e86e59d82f19205fb9d83a1588a39aa78a7f1ba966b286e89c99b3d04d84ee2"} Oct 30 21:55:23.829063 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:23.828595 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4fhws" podStartSLOduration=1.828575844 podStartE2EDuration="1.828575844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:23.825018017 +0000 UTC m=+889.995983202" watchObservedRunningTime="2026-10-30 21:55:23.828575844 +0000 UTC m=+889.999541026" Oct 30 21:55:24.319769 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:24.319742 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4fhws" Oct 30 21:55:24.321169 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:24.321146 6720 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4fhws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 21:55:24.321340 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:24.321189 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4fhws" podUID="3e578cfa-7359-4c08-a8d0-56989d62227d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 21:55:24.376605 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:24.367949 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:25.140727 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:25.140685 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:25.319672 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:25.319632 6720 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4fhws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 21:55:25.319869 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:25.319696 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4fhws" podUID="3e578cfa-7359-4c08-a8d0-56989d62227d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 21:55:25.383548 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:25.383521 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:26.319647 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:26.319610 6720 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4fhws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 21:55:26.319821 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:26.319677 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4fhws" podUID="3e578cfa-7359-4c08-a8d0-56989d62227d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 21:55:26.408366 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:26.408325 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:26.677409 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:26.675840 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:27.287083 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:27.287046 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-689a2563/e2e-lifecycle-model-689a2563-kserve-5bb4c5b87c-q6gl8"] Oct 30 21:55:27.292923 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:27.292897 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-689a2563/e2e-lifecycle-model-689a2563-kserve-5bb4c5b87c-q6gl8"] Oct 30 21:55:27.293235 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:27.293218 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-689a2563/e2e-lifecycle-model-689a2563-kserve-5bb4c5b87c-q6gl8" Oct 30 21:55:27.310426 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:27.308938 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-689a2563\"/\"e2e-lifecycle-model-689a2563-kserve-self-signed-certs\"" Oct 30 21:55:27.310426 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:27.309098 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-689a2563\"/\"default-dockercfg-wc7l5\"" Oct 30 21:55:27.310614 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:27.310443 6720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-689a2563\"/\"openshift-service-ca.crt\"" Oct 30 21:55:27.310799 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:27.310757 6720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-689a2563\"/\"kube-root-ca.crt\"" Oct 30 21:55:27.319683 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:27.319658 6720 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4fhws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 21:55:27.319801 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:27.319704 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4fhws" podUID="3e578cfa-7359-4c08-a8d0-56989d62227d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 21:55:27.380365 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:27.380333 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:27.382685 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:27.382651 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/450c5ec5-0a77-449d-a515-1aa21776a572-tmp-dir\") pod \"e2e-lifecycle-model-689a2563-kserve-5bb4c5b87c-q6gl8\" (UID: \"450c5ec5-0a77-449d-a515-1aa21776a572\") " pod="ai-tenant-e2e-mt-689a2563/e2e-lifecycle-model-689a2563-kserve-5bb4c5b87c-q6gl8" Oct 30 21:55:27.382849 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:27.382724 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/450c5ec5-0a77-449d-a515-1aa21776a572-tls-certs\") pod \"e2e-lifecycle-model-689a2563-kserve-5bb4c5b87c-q6gl8\" (UID: \"450c5ec5-0a77-449d-a515-1aa21776a572\") " pod="ai-tenant-e2e-mt-689a2563/e2e-lifecycle-model-689a2563-kserve-5bb4c5b87c-q6gl8" Oct 30 21:55:27.382849 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:27.382768 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/450c5ec5-0a77-449d-a515-1aa21776a572-home\") pod \"e2e-lifecycle-model-689a2563-kserve-5bb4c5b87c-q6gl8\" (UID: \"450c5ec5-0a77-449d-a515-1aa21776a572\") " pod="ai-tenant-e2e-mt-689a2563/e2e-lifecycle-model-689a2563-kserve-5bb4c5b87c-q6gl8" Oct 30 21:55:27.382849 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:27.382820 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/450c5ec5-0a77-449d-a515-1aa21776a572-model-cache\") pod \"e2e-lifecycle-model-689a2563-kserve-5bb4c5b87c-q6gl8\" (UID: \"450c5ec5-0a77-449d-a515-1aa21776a572\") " pod="ai-tenant-e2e-mt-689a2563/e2e-lifecycle-model-689a2563-kserve-5bb4c5b87c-q6gl8" Oct 30 21:55:27.383037 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:27.383019 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvq7n\" (UniqueName: \"kubernetes.io/projected/450c5ec5-0a77-449d-a515-1aa21776a572-kube-api-access-vvq7n\") pod \"e2e-lifecycle-model-689a2563-kserve-5bb4c5b87c-q6gl8\" (UID: \"450c5ec5-0a77-449d-a515-1aa21776a572\") " pod="ai-tenant-e2e-mt-689a2563/e2e-lifecycle-model-689a2563-kserve-5bb4c5b87c-q6gl8" Oct 30 21:55:27.383233 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:27.383211 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/450c5ec5-0a77-449d-a515-1aa21776a572-dshm\") pod \"e2e-lifecycle-model-689a2563-kserve-5bb4c5b87c-q6gl8\" (UID: \"450c5ec5-0a77-449d-a515-1aa21776a572\") " pod="ai-tenant-e2e-mt-689a2563/e2e-lifecycle-model-689a2563-kserve-5bb4c5b87c-q6gl8" Oct 30 21:55:27.489500 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:27.487208 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/450c5ec5-0a77-449d-a515-1aa21776a572-tmp-dir\") pod \"e2e-lifecycle-model-689a2563-kserve-5bb4c5b87c-q6gl8\" (UID: \"450c5ec5-0a77-449d-a515-1aa21776a572\") " pod="ai-tenant-e2e-mt-689a2563/e2e-lifecycle-model-689a2563-kserve-5bb4c5b87c-q6gl8" Oct 30 21:55:27.489500 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:27.487291 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/450c5ec5-0a77-449d-a515-1aa21776a572-tls-certs\") pod \"e2e-lifecycle-model-689a2563-kserve-5bb4c5b87c-q6gl8\" (UID: \"450c5ec5-0a77-449d-a515-1aa21776a572\") " pod="ai-tenant-e2e-mt-689a2563/e2e-lifecycle-model-689a2563-kserve-5bb4c5b87c-q6gl8" Oct 30 21:55:27.489500 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:27.487330 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/450c5ec5-0a77-449d-a515-1aa21776a572-home\") pod \"e2e-lifecycle-model-689a2563-kserve-5bb4c5b87c-q6gl8\" (UID: \"450c5ec5-0a77-449d-a515-1aa21776a572\") " pod="ai-tenant-e2e-mt-689a2563/e2e-lifecycle-model-689a2563-kserve-5bb4c5b87c-q6gl8" Oct 30 21:55:27.489500 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:27.487372 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/450c5ec5-0a77-449d-a515-1aa21776a572-model-cache\") pod \"e2e-lifecycle-model-689a2563-kserve-5bb4c5b87c-q6gl8\" (UID: \"450c5ec5-0a77-449d-a515-1aa21776a572\") " pod="ai-tenant-e2e-mt-689a2563/e2e-lifecycle-model-689a2563-kserve-5bb4c5b87c-q6gl8" Oct 30 21:55:27.489500 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:27.487522 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvq7n\" (UniqueName: \"kubernetes.io/projected/450c5ec5-0a77-449d-a515-1aa21776a572-kube-api-access-vvq7n\") pod \"e2e-lifecycle-model-689a2563-kserve-5bb4c5b87c-q6gl8\" (UID: \"450c5ec5-0a77-449d-a515-1aa21776a572\") " pod="ai-tenant-e2e-mt-689a2563/e2e-lifecycle-model-689a2563-kserve-5bb4c5b87c-q6gl8" Oct 30 21:55:27.489500 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:27.487661 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/450c5ec5-0a77-449d-a515-1aa21776a572-dshm\") pod \"e2e-lifecycle-model-689a2563-kserve-5bb4c5b87c-q6gl8\" (UID: \"450c5ec5-0a77-449d-a515-1aa21776a572\") " pod="ai-tenant-e2e-mt-689a2563/e2e-lifecycle-model-689a2563-kserve-5bb4c5b87c-q6gl8" Oct 30 21:55:27.489500 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:27.488481 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/450c5ec5-0a77-449d-a515-1aa21776a572-home\") pod \"e2e-lifecycle-model-689a2563-kserve-5bb4c5b87c-q6gl8\" (UID: \"450c5ec5-0a77-449d-a515-1aa21776a572\") " pod="ai-tenant-e2e-mt-689a2563/e2e-lifecycle-model-689a2563-kserve-5bb4c5b87c-q6gl8" Oct 30 21:55:27.489500 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:27.488765 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/450c5ec5-0a77-449d-a515-1aa21776a572-model-cache\") pod \"e2e-lifecycle-model-689a2563-kserve-5bb4c5b87c-q6gl8\" (UID: \"450c5ec5-0a77-449d-a515-1aa21776a572\") " pod="ai-tenant-e2e-mt-689a2563/e2e-lifecycle-model-689a2563-kserve-5bb4c5b87c-q6gl8" Oct 30 21:55:27.519317 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:27.519283 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/450c5ec5-0a77-449d-a515-1aa21776a572-tmp-dir\") pod \"e2e-lifecycle-model-689a2563-kserve-5bb4c5b87c-q6gl8\" (UID: \"450c5ec5-0a77-449d-a515-1aa21776a572\") " pod="ai-tenant-e2e-mt-689a2563/e2e-lifecycle-model-689a2563-kserve-5bb4c5b87c-q6gl8" Oct 30 21:55:27.533341 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:27.533302 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/450c5ec5-0a77-449d-a515-1aa21776a572-tls-certs\") pod \"e2e-lifecycle-model-689a2563-kserve-5bb4c5b87c-q6gl8\" (UID: \"450c5ec5-0a77-449d-a515-1aa21776a572\") " pod="ai-tenant-e2e-mt-689a2563/e2e-lifecycle-model-689a2563-kserve-5bb4c5b87c-q6gl8" Oct 30 21:55:27.533950 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:27.533925 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/450c5ec5-0a77-449d-a515-1aa21776a572-dshm\") pod \"e2e-lifecycle-model-689a2563-kserve-5bb4c5b87c-q6gl8\" (UID: \"450c5ec5-0a77-449d-a515-1aa21776a572\") " pod="ai-tenant-e2e-mt-689a2563/e2e-lifecycle-model-689a2563-kserve-5bb4c5b87c-q6gl8" Oct 30 21:55:27.561900 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:27.561762 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:27.587136 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:27.587059 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvq7n\" (UniqueName: \"kubernetes.io/projected/450c5ec5-0a77-449d-a515-1aa21776a572-kube-api-access-vvq7n\") pod \"e2e-lifecycle-model-689a2563-kserve-5bb4c5b87c-q6gl8\" (UID: \"450c5ec5-0a77-449d-a515-1aa21776a572\") " pod="ai-tenant-e2e-mt-689a2563/e2e-lifecycle-model-689a2563-kserve-5bb4c5b87c-q6gl8" Oct 30 21:55:27.619410 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:27.618485 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-689a2563/e2e-lifecycle-model-689a2563-kserve-5bb4c5b87c-q6gl8" Oct 30 21:55:27.975439 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:55:27.975407 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod450c5ec5_0a77_449d_a515_1aa21776a572.slice/crio-4e4c9b43bd7485c9afa6233ffebf0ec06c8be06e90c581c45e241de8e02d36ef WatchSource:0}: Error finding container 4e4c9b43bd7485c9afa6233ffebf0ec06c8be06e90c581c45e241de8e02d36ef: Status 404 returned error can't find the container with id 4e4c9b43bd7485c9afa6233ffebf0ec06c8be06e90c581c45e241de8e02d36ef Oct 30 21:55:27.978924 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:27.978905 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-689a2563/e2e-lifecycle-model-689a2563-kserve-5bb4c5b87c-q6gl8"] Oct 30 21:55:28.320215 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:28.320146 6720 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4fhws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 21:55:28.320480 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:28.320456 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4fhws" podUID="3e578cfa-7359-4c08-a8d0-56989d62227d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 21:55:28.487406 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:28.482373 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:28.820659 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:28.820622 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-689a2563/e2e-lifecycle-model-689a2563-kserve-5bb4c5b87c-q6gl8" event={"ID":"450c5ec5-0a77-449d-a515-1aa21776a572","Type":"ContainerStarted","Data":"a2280742c1687520374c40f9e9eb2b7a0374bddacb7a65d6fef0d5b788a23851"} Oct 30 21:55:28.821031 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:28.820668 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-689a2563/e2e-lifecycle-model-689a2563-kserve-5bb4c5b87c-q6gl8" event={"ID":"450c5ec5-0a77-449d-a515-1aa21776a572","Type":"ContainerStarted","Data":"4e4c9b43bd7485c9afa6233ffebf0ec06c8be06e90c581c45e241de8e02d36ef"} Oct 30 21:55:28.870433 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:28.864291 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-689a2563/e2e-lifecycle-model-689a2563-kserve-5bb4c5b87c-q6gl8" podStartSLOduration=1.864271139 podStartE2EDuration="1.864271139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:28.861779473 +0000 UTC m=+895.032744654" watchObservedRunningTime="2026-10-30 21:55:28.864271139 +0000 UTC m=+895.035236322" Oct 30 21:55:29.191591 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:29.191562 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:29.319775 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:29.319738 6720 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4fhws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 21:55:29.319968 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:29.319782 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4fhws" podUID="3e578cfa-7359-4c08-a8d0-56989d62227d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 21:55:29.618983 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:29.618951 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-689a2563/e2e-lifecycle-model-689a2563-kserve-5bb4c5b87c-q6gl8" Oct 30 21:55:29.946724 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:29.946646 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:30.133780 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:30.133727 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:30.319945 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:30.319860 6720 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4fhws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 21:55:30.319945 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:30.319920 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4fhws" podUID="3e578cfa-7359-4c08-a8d0-56989d62227d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 21:55:30.566455 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:30.566400 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:30.852193 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:30.852151 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:31.325206 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:31.325174 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4fhws" Oct 30 21:55:31.326594 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:31.326567 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4fhws" Oct 30 21:55:31.330732 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:31.330694 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4fhws" Oct 30 21:55:31.334202 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:31.334170 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:32.056303 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:32.056260 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:33.027827 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:33.027792 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:34.380347 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:34.380319 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:34.665373 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:34.665266 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:35.975027 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:35.974989 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:36.283166 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:36.283086 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:37.142741 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:37.142709 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:37.619081 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:37.619051 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-689a2563/e2e-lifecycle-model-689a2563-kserve-5bb4c5b87c-q6gl8" Oct 30 21:55:37.632775 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:37.632747 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-689a2563/e2e-lifecycle-model-689a2563-kserve-5bb4c5b87c-q6gl8" Oct 30 21:55:37.942484 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:37.942359 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:37.945664 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:37.945640 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-689a2563/e2e-lifecycle-model-689a2563-kserve-5bb4c5b87c-q6gl8" Oct 30 21:55:38.975416 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:38.975370 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:39.906271 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:39.906237 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:40.404955 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:55:40.404910 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 21:55:40.540898 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:40.540861 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:41.125256 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:41.125223 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2c9c95-55d58b4c57-hk5bq"] Oct 30 21:55:41.131962 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:41.131939 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c9c95-55d58b4c57-hk5bq" Oct 30 21:55:41.139035 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:41.139008 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-2c9c95-serving-cert\"" Oct 30 21:55:41.146983 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:41.146960 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2c9c95-55d58b4c57-hk5bq"] Oct 30 21:55:41.245001 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:41.244970 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:41.266275 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:41.266246 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5njxf\" (UniqueName: \"kubernetes.io/projected/43a8441e-d4cc-4802-a621-ffdbe0e13e0b-kube-api-access-5njxf\") pod \"maas-api-e2e-api-a-2c9c95-55d58b4c57-hk5bq\" (UID: \"43a8441e-d4cc-4802-a621-ffdbe0e13e0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c9c95-55d58b4c57-hk5bq" Oct 30 21:55:41.266469 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:41.266303 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43a8441e-d4cc-4802-a621-ffdbe0e13e0b-metrics-certs\") pod \"maas-api-e2e-api-a-2c9c95-55d58b4c57-hk5bq\" (UID: \"43a8441e-d4cc-4802-a621-ffdbe0e13e0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c9c95-55d58b4c57-hk5bq" Oct 30 21:55:41.266528 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:41.266475 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43a8441e-d4cc-4802-a621-ffdbe0e13e0b-maas-api-tls\") pod \"maas-api-e2e-api-a-2c9c95-55d58b4c57-hk5bq\" (UID: \"43a8441e-d4cc-4802-a621-ffdbe0e13e0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c9c95-55d58b4c57-hk5bq" Oct 30 21:55:41.266598 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:41.266564 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43a8441e-d4cc-4802-a621-ffdbe0e13e0b-openshift-service-ca\") pod \"maas-api-e2e-api-a-2c9c95-55d58b4c57-hk5bq\" (UID: \"43a8441e-d4cc-4802-a621-ffdbe0e13e0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c9c95-55d58b4c57-hk5bq" Oct 30 21:55:41.367828 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:41.367798 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5njxf\" (UniqueName: \"kubernetes.io/projected/43a8441e-d4cc-4802-a621-ffdbe0e13e0b-kube-api-access-5njxf\") pod \"maas-api-e2e-api-a-2c9c95-55d58b4c57-hk5bq\" (UID: \"43a8441e-d4cc-4802-a621-ffdbe0e13e0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c9c95-55d58b4c57-hk5bq" Oct 30 21:55:41.368008 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:41.367839 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43a8441e-d4cc-4802-a621-ffdbe0e13e0b-metrics-certs\") pod \"maas-api-e2e-api-a-2c9c95-55d58b4c57-hk5bq\" (UID: \"43a8441e-d4cc-4802-a621-ffdbe0e13e0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c9c95-55d58b4c57-hk5bq" Oct 30 21:55:41.368008 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:41.367878 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43a8441e-d4cc-4802-a621-ffdbe0e13e0b-maas-api-tls\") pod \"maas-api-e2e-api-a-2c9c95-55d58b4c57-hk5bq\" (UID: \"43a8441e-d4cc-4802-a621-ffdbe0e13e0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c9c95-55d58b4c57-hk5bq" Oct 30 21:55:41.368008 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:41.367923 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43a8441e-d4cc-4802-a621-ffdbe0e13e0b-openshift-service-ca\") pod \"maas-api-e2e-api-a-2c9c95-55d58b4c57-hk5bq\" (UID: \"43a8441e-d4cc-4802-a621-ffdbe0e13e0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c9c95-55d58b4c57-hk5bq" Oct 30 21:55:41.368589 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:41.368566 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43a8441e-d4cc-4802-a621-ffdbe0e13e0b-openshift-service-ca\") pod \"maas-api-e2e-api-a-2c9c95-55d58b4c57-hk5bq\" (UID: \"43a8441e-d4cc-4802-a621-ffdbe0e13e0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c9c95-55d58b4c57-hk5bq" Oct 30 21:55:41.370319 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:41.370301 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43a8441e-d4cc-4802-a621-ffdbe0e13e0b-metrics-certs\") pod \"maas-api-e2e-api-a-2c9c95-55d58b4c57-hk5bq\" (UID: \"43a8441e-d4cc-4802-a621-ffdbe0e13e0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c9c95-55d58b4c57-hk5bq" Oct 30 21:55:41.370450 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:41.370433 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43a8441e-d4cc-4802-a621-ffdbe0e13e0b-maas-api-tls\") pod \"maas-api-e2e-api-a-2c9c95-55d58b4c57-hk5bq\" (UID: \"43a8441e-d4cc-4802-a621-ffdbe0e13e0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c9c95-55d58b4c57-hk5bq" Oct 30 21:55:41.381151 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:41.381082 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5njxf\" (UniqueName: \"kubernetes.io/projected/43a8441e-d4cc-4802-a621-ffdbe0e13e0b-kube-api-access-5njxf\") pod \"maas-api-e2e-api-a-2c9c95-55d58b4c57-hk5bq\" (UID: \"43a8441e-d4cc-4802-a621-ffdbe0e13e0b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c9c95-55d58b4c57-hk5bq" Oct 30 21:55:41.427363 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:41.427339 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:41.449476 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:41.449452 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c9c95-55d58b4c57-hk5bq" Oct 30 21:55:41.732897 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:41.732872 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2c9c95-55d58b4c57-hk5bq"] Oct 30 21:55:41.734104 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:55:41.734078 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43a8441e_d4cc_4802_a621_ffdbe0e13e0b.slice/crio-f072dff4a331295e06015e43c8df1d81221607721324781436d78f353f13aba4 WatchSource:0}: Error finding container f072dff4a331295e06015e43c8df1d81221607721324781436d78f353f13aba4: Status 404 returned error can't find the container with id f072dff4a331295e06015e43c8df1d81221607721324781436d78f353f13aba4 Oct 30 21:55:41.969411 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:41.968187 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c9c95-55d58b4c57-hk5bq" event={"ID":"43a8441e-d4cc-4802-a621-ffdbe0e13e0b","Type":"ContainerStarted","Data":"f072dff4a331295e06015e43c8df1d81221607721324781436d78f353f13aba4"} Oct 30 21:55:42.210277 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:42.210242 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:42.977614 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:42.977580 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c9c95-55d58b4c57-hk5bq" event={"ID":"43a8441e-d4cc-4802-a621-ffdbe0e13e0b","Type":"ContainerStarted","Data":"87bad815139a4f47f51b85a63a700b1d03d7e5c15192bf71332a22ecdf52c79a"} Oct 30 21:55:42.978023 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:42.977652 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c9c95-55d58b4c57-hk5bq" Oct 30 21:55:42.999356 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:42.999311 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c9c95-55d58b4c57-hk5bq" podStartSLOduration=1.5450011030000002 podStartE2EDuration="1.99929626s" totalImagesPullingTime="454.295157ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:42.997334817 +0000 UTC m=+909.168299999" watchObservedRunningTime="2026-10-30 21:55:42.99929626 +0000 UTC m=+909.170261495" Oct 30 21:55:43.024992 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:43.024962 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:43.723008 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:43.722927 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:44.421812 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:44.421772 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:45.188884 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:45.188852 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:45.902169 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:45.902137 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:46.545786 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:46.545753 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:47.224939 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:47.224691 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:47.887690 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:47.887657 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:48.999632 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:48.999588 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c9c95-55d58b4c57-hk5bq" Oct 30 21:55:49.005459 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:49.005356 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:49.859566 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:49.859527 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:50.623000 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:50.622958 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:51.297193 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:51.297163 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:51.960814 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:51.960785 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:52.773785 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:52.773755 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:53.521833 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:53.521799 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:54.304085 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:54.304048 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:55.136986 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:55.136956 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:56.083771 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:56.083735 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:56.110773 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:56.110743 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-mvxk6"] Oct 30 21:55:56.114863 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:56.114842 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-mvxk6" Oct 30 21:55:56.134470 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:56.134445 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-mvxk6"] Oct 30 21:55:56.267360 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:56.267326 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d0a4f90-fa48-4e0e-b6fa-167055c872f1-istiod-ca-cert\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-mvxk6\" (UID: \"8d0a4f90-fa48-4e0e-b6fa-167055c872f1\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-mvxk6" Oct 30 21:55:56.267360 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:56.267360 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7mbx\" (UniqueName: \"kubernetes.io/projected/8d0a4f90-fa48-4e0e-b6fa-167055c872f1-kube-api-access-n7mbx\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-mvxk6\" (UID: \"8d0a4f90-fa48-4e0e-b6fa-167055c872f1\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-mvxk6" Oct 30 21:55:56.267960 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:56.267500 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d0a4f90-fa48-4e0e-b6fa-167055c872f1-istio-envoy\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-mvxk6\" (UID: \"8d0a4f90-fa48-4e0e-b6fa-167055c872f1\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-mvxk6" Oct 30 21:55:56.267960 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:56.267652 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d0a4f90-fa48-4e0e-b6fa-167055c872f1-workload-socket\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-mvxk6\" (UID: \"8d0a4f90-fa48-4e0e-b6fa-167055c872f1\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-mvxk6" Oct 30 21:55:56.267960 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:56.267682 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d0a4f90-fa48-4e0e-b6fa-167055c872f1-istio-data\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-mvxk6\" (UID: \"8d0a4f90-fa48-4e0e-b6fa-167055c872f1\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-mvxk6" Oct 30 21:55:56.267960 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:56.267703 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d0a4f90-fa48-4e0e-b6fa-167055c872f1-istio-token\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-mvxk6\" (UID: \"8d0a4f90-fa48-4e0e-b6fa-167055c872f1\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-mvxk6" Oct 30 21:55:56.267960 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:56.267726 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d0a4f90-fa48-4e0e-b6fa-167055c872f1-workload-certs\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-mvxk6\" (UID: \"8d0a4f90-fa48-4e0e-b6fa-167055c872f1\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-mvxk6" Oct 30 21:55:56.267960 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:56.267815 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d0a4f90-fa48-4e0e-b6fa-167055c872f1-credential-socket\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-mvxk6\" (UID: \"8d0a4f90-fa48-4e0e-b6fa-167055c872f1\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-mvxk6" Oct 30 21:55:56.267960 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:56.267906 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d0a4f90-fa48-4e0e-b6fa-167055c872f1-istio-podinfo\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-mvxk6\" (UID: \"8d0a4f90-fa48-4e0e-b6fa-167055c872f1\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-mvxk6" Oct 30 21:55:56.368820 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:56.368791 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d0a4f90-fa48-4e0e-b6fa-167055c872f1-credential-socket\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-mvxk6\" (UID: \"8d0a4f90-fa48-4e0e-b6fa-167055c872f1\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-mvxk6" Oct 30 21:55:56.368993 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:56.368869 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d0a4f90-fa48-4e0e-b6fa-167055c872f1-istio-podinfo\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-mvxk6\" (UID: \"8d0a4f90-fa48-4e0e-b6fa-167055c872f1\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-mvxk6" Oct 30 21:55:56.368993 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:56.368918 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d0a4f90-fa48-4e0e-b6fa-167055c872f1-istiod-ca-cert\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-mvxk6\" (UID: \"8d0a4f90-fa48-4e0e-b6fa-167055c872f1\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-mvxk6" Oct 30 21:55:56.368993 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:56.368934 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7mbx\" (UniqueName: \"kubernetes.io/projected/8d0a4f90-fa48-4e0e-b6fa-167055c872f1-kube-api-access-n7mbx\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-mvxk6\" (UID: \"8d0a4f90-fa48-4e0e-b6fa-167055c872f1\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-mvxk6" Oct 30 21:55:56.368993 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:56.368986 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d0a4f90-fa48-4e0e-b6fa-167055c872f1-istio-envoy\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-mvxk6\" (UID: \"8d0a4f90-fa48-4e0e-b6fa-167055c872f1\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-mvxk6" Oct 30 21:55:56.369219 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:56.369065 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d0a4f90-fa48-4e0e-b6fa-167055c872f1-workload-socket\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-mvxk6\" (UID: \"8d0a4f90-fa48-4e0e-b6fa-167055c872f1\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-mvxk6" Oct 30 21:55:56.369219 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:56.369094 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d0a4f90-fa48-4e0e-b6fa-167055c872f1-istio-data\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-mvxk6\" (UID: \"8d0a4f90-fa48-4e0e-b6fa-167055c872f1\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-mvxk6" Oct 30 21:55:56.369219 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:56.369120 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d0a4f90-fa48-4e0e-b6fa-167055c872f1-istio-token\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-mvxk6\" (UID: \"8d0a4f90-fa48-4e0e-b6fa-167055c872f1\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-mvxk6" Oct 30 21:55:56.369219 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:56.369153 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d0a4f90-fa48-4e0e-b6fa-167055c872f1-workload-certs\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-mvxk6\" (UID: \"8d0a4f90-fa48-4e0e-b6fa-167055c872f1\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-mvxk6" Oct 30 21:55:56.369480 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:56.369241 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d0a4f90-fa48-4e0e-b6fa-167055c872f1-credential-socket\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-mvxk6\" (UID: \"8d0a4f90-fa48-4e0e-b6fa-167055c872f1\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-mvxk6" Oct 30 21:55:56.369575 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:56.369552 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d0a4f90-fa48-4e0e-b6fa-167055c872f1-workload-certs\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-mvxk6\" (UID: \"8d0a4f90-fa48-4e0e-b6fa-167055c872f1\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-mvxk6" Oct 30 21:55:56.369644 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:56.369585 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d0a4f90-fa48-4e0e-b6fa-167055c872f1-istio-data\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-mvxk6\" (UID: \"8d0a4f90-fa48-4e0e-b6fa-167055c872f1\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-mvxk6" Oct 30 21:55:56.369644 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:56.369633 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d0a4f90-fa48-4e0e-b6fa-167055c872f1-istiod-ca-cert\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-mvxk6\" (UID: \"8d0a4f90-fa48-4e0e-b6fa-167055c872f1\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-mvxk6" Oct 30 21:55:56.369920 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:56.369793 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d0a4f90-fa48-4e0e-b6fa-167055c872f1-workload-socket\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-mvxk6\" (UID: \"8d0a4f90-fa48-4e0e-b6fa-167055c872f1\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-mvxk6" Oct 30 21:55:56.371567 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:56.371547 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d0a4f90-fa48-4e0e-b6fa-167055c872f1-istio-envoy\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-mvxk6\" (UID: \"8d0a4f90-fa48-4e0e-b6fa-167055c872f1\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-mvxk6" Oct 30 21:55:56.371718 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:56.371696 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d0a4f90-fa48-4e0e-b6fa-167055c872f1-istio-podinfo\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-mvxk6\" (UID: \"8d0a4f90-fa48-4e0e-b6fa-167055c872f1\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-mvxk6" Oct 30 21:55:56.386069 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:56.386044 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7mbx\" (UniqueName: \"kubernetes.io/projected/8d0a4f90-fa48-4e0e-b6fa-167055c872f1-kube-api-access-n7mbx\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-mvxk6\" (UID: \"8d0a4f90-fa48-4e0e-b6fa-167055c872f1\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-mvxk6" Oct 30 21:55:56.387133 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:56.387112 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d0a4f90-fa48-4e0e-b6fa-167055c872f1-istio-token\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-mvxk6\" (UID: \"8d0a4f90-fa48-4e0e-b6fa-167055c872f1\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-mvxk6" Oct 30 21:55:56.429103 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:56.429076 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-mvxk6" Oct 30 21:55:56.729378 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:55:56.729337 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d0a4f90_fa48_4e0e_b6fa_167055c872f1.slice/crio-d17cbc5bdc9665d671ac6947c8fea50e218d2c243cdc7b0f22eece2108e32d42 WatchSource:0}: Error finding container d17cbc5bdc9665d671ac6947c8fea50e218d2c243cdc7b0f22eece2108e32d42: Status 404 returned error can't find the container with id d17cbc5bdc9665d671ac6947c8fea50e218d2c243cdc7b0f22eece2108e32d42 Oct 30 21:55:56.731315 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:56.731196 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-mvxk6"] Oct 30 21:55:56.732176 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:56.732141 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:55:56.732246 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:56.732209 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:55:56.732246 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:56.732237 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:55:56.916114 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:56.916073 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:57.073698 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:57.073602 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-mvxk6" event={"ID":"8d0a4f90-fa48-4e0e-b6fa-167055c872f1","Type":"ContainerStarted","Data":"b523f2bb8ac7f1f156be990df6e687253d26332aa3b81c6df9589ddba2ebb712"} Oct 30 21:55:57.073698 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:57.073657 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-mvxk6" event={"ID":"8d0a4f90-fa48-4e0e-b6fa-167055c872f1","Type":"ContainerStarted","Data":"d17cbc5bdc9665d671ac6947c8fea50e218d2c243cdc7b0f22eece2108e32d42"} Oct 30 21:55:57.101328 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:57.101283 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-mvxk6" podStartSLOduration=1.101268696 podStartE2EDuration="1.101268696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:57.099574048 +0000 UTC m=+923.270539231" watchObservedRunningTime="2026-10-30 21:55:57.101268696 +0000 UTC m=+923.272233878" Oct 30 21:55:57.429718 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:57.429689 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-mvxk6" Oct 30 21:55:57.431422 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:57.431372 6720 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-mvxk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 21:55:57.431538 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:57.431447 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-mvxk6" podUID="8d0a4f90-fa48-4e0e-b6fa-167055c872f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 21:55:57.632095 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:57.631734 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:58.429976 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:58.429940 6720 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-mvxk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 21:55:58.430436 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:58.430006 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-mvxk6" podUID="8d0a4f90-fa48-4e0e-b6fa-167055c872f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 21:55:58.434669 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:58.434642 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:59.158416 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:59.149770 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:59.429698 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:59.429622 6720 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-mvxk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 21:55:59.429698 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:59.429689 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-mvxk6" podUID="8d0a4f90-fa48-4e0e-b6fa-167055c872f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 21:55:59.732510 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:59.732432 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:55:59.895002 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:55:59.894971 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:00.429555 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:00.429523 6720 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-mvxk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 21:56:00.429734 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:00.429579 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-mvxk6" podUID="8d0a4f90-fa48-4e0e-b6fa-167055c872f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 21:56:00.494081 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:00.494041 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:00.890407 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:00.887126 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:01.121411 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:01.120894 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:01.380404 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:01.380260 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:01.429363 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:01.429328 6720 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-mvxk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 21:56:01.429540 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:01.429375 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-mvxk6" podUID="8d0a4f90-fa48-4e0e-b6fa-167055c872f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 21:56:01.885224 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:01.885193 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:02.432259 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:02.431786 6720 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-mvxk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 21:56:02.432259 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:02.431846 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-mvxk6" podUID="8d0a4f90-fa48-4e0e-b6fa-167055c872f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 21:56:02.770019 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:02.769935 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:03.429997 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:03.429969 6720 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-mvxk6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 21:56:03.430167 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:03.430024 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-mvxk6" podUID="8d0a4f90-fa48-4e0e-b6fa-167055c872f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 21:56:03.839926 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:03.839610 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:04.326498 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:04.326426 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:04.433758 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:04.433731 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-mvxk6" Oct 30 21:56:04.434154 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:04.434131 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-mvxk6" Oct 30 21:56:04.434927 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:04.434905 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-mvxk6" Oct 30 21:56:04.489794 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:04.489764 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:04.827672 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:04.827637 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:05.548130 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:05.548098 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:05.751755 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:05.751721 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:06.540575 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:06.540535 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:07.425047 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:07.425013 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:08.121961 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:08.121920 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:08.909737 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:08.909708 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:09.550770 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:09.550731 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:10.300215 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:10.300186 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:10.408182 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:56:10.408146 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 21:56:10.458052 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:10.458001 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:11.161492 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:11.161456 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:11.343040 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:11.343009 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jqtlj"] Oct 30 21:56:11.347496 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:11.347472 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jqtlj" Oct 30 21:56:11.368059 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:11.368025 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jqtlj"] Oct 30 21:56:11.491352 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:11.491279 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc5329cd-398f-42a3-8e4d-82d3b6446773-istiod-ca-cert\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jqtlj\" (UID: \"fc5329cd-398f-42a3-8e4d-82d3b6446773\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jqtlj" Oct 30 21:56:11.491513 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:11.491467 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc5329cd-398f-42a3-8e4d-82d3b6446773-workload-certs\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jqtlj\" (UID: \"fc5329cd-398f-42a3-8e4d-82d3b6446773\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jqtlj" Oct 30 21:56:11.491513 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:11.491502 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc5329cd-398f-42a3-8e4d-82d3b6446773-credential-socket\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jqtlj\" (UID: \"fc5329cd-398f-42a3-8e4d-82d3b6446773\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jqtlj" Oct 30 21:56:11.491625 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:11.491527 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stws9\" (UniqueName: \"kubernetes.io/projected/fc5329cd-398f-42a3-8e4d-82d3b6446773-kube-api-access-stws9\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jqtlj\" (UID: \"fc5329cd-398f-42a3-8e4d-82d3b6446773\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jqtlj" Oct 30 21:56:11.491708 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:11.491685 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc5329cd-398f-42a3-8e4d-82d3b6446773-istio-envoy\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jqtlj\" (UID: \"fc5329cd-398f-42a3-8e4d-82d3b6446773\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jqtlj" Oct 30 21:56:11.491768 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:11.491727 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc5329cd-398f-42a3-8e4d-82d3b6446773-workload-socket\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jqtlj\" (UID: \"fc5329cd-398f-42a3-8e4d-82d3b6446773\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jqtlj" Oct 30 21:56:11.491854 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:11.491841 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc5329cd-398f-42a3-8e4d-82d3b6446773-istio-podinfo\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jqtlj\" (UID: \"fc5329cd-398f-42a3-8e4d-82d3b6446773\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jqtlj" Oct 30 21:56:11.491916 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:11.491866 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc5329cd-398f-42a3-8e4d-82d3b6446773-istio-token\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jqtlj\" (UID: \"fc5329cd-398f-42a3-8e4d-82d3b6446773\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jqtlj" Oct 30 21:56:11.491916 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:11.491897 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc5329cd-398f-42a3-8e4d-82d3b6446773-istio-data\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jqtlj\" (UID: \"fc5329cd-398f-42a3-8e4d-82d3b6446773\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jqtlj" Oct 30 21:56:11.592684 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:11.592652 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc5329cd-398f-42a3-8e4d-82d3b6446773-istio-data\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jqtlj\" (UID: \"fc5329cd-398f-42a3-8e4d-82d3b6446773\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jqtlj" Oct 30 21:56:11.592853 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:11.592729 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc5329cd-398f-42a3-8e4d-82d3b6446773-istiod-ca-cert\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jqtlj\" (UID: \"fc5329cd-398f-42a3-8e4d-82d3b6446773\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jqtlj" Oct 30 21:56:11.592902 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:11.592872 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc5329cd-398f-42a3-8e4d-82d3b6446773-workload-certs\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jqtlj\" (UID: \"fc5329cd-398f-42a3-8e4d-82d3b6446773\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jqtlj" Oct 30 21:56:11.592946 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:11.592913 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc5329cd-398f-42a3-8e4d-82d3b6446773-credential-socket\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jqtlj\" (UID: \"fc5329cd-398f-42a3-8e4d-82d3b6446773\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jqtlj" Oct 30 21:56:11.592983 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:11.592950 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stws9\" (UniqueName: \"kubernetes.io/projected/fc5329cd-398f-42a3-8e4d-82d3b6446773-kube-api-access-stws9\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jqtlj\" (UID: \"fc5329cd-398f-42a3-8e4d-82d3b6446773\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jqtlj" Oct 30 21:56:11.593038 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:11.593020 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc5329cd-398f-42a3-8e4d-82d3b6446773-istio-envoy\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jqtlj\" (UID: \"fc5329cd-398f-42a3-8e4d-82d3b6446773\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jqtlj" Oct 30 21:56:11.593094 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:11.593052 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc5329cd-398f-42a3-8e4d-82d3b6446773-workload-socket\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jqtlj\" (UID: \"fc5329cd-398f-42a3-8e4d-82d3b6446773\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jqtlj" Oct 30 21:56:11.593150 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:11.593098 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc5329cd-398f-42a3-8e4d-82d3b6446773-istio-data\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jqtlj\" (UID: \"fc5329cd-398f-42a3-8e4d-82d3b6446773\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jqtlj" Oct 30 21:56:11.593150 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:11.593122 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc5329cd-398f-42a3-8e4d-82d3b6446773-istio-podinfo\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jqtlj\" (UID: \"fc5329cd-398f-42a3-8e4d-82d3b6446773\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jqtlj" Oct 30 21:56:11.593240 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:11.593145 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc5329cd-398f-42a3-8e4d-82d3b6446773-istio-token\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jqtlj\" (UID: \"fc5329cd-398f-42a3-8e4d-82d3b6446773\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jqtlj" Oct 30 21:56:11.593444 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:11.593336 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc5329cd-398f-42a3-8e4d-82d3b6446773-credential-socket\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jqtlj\" (UID: \"fc5329cd-398f-42a3-8e4d-82d3b6446773\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jqtlj" Oct 30 21:56:11.593564 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:11.593511 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc5329cd-398f-42a3-8e4d-82d3b6446773-istiod-ca-cert\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jqtlj\" (UID: \"fc5329cd-398f-42a3-8e4d-82d3b6446773\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jqtlj" Oct 30 21:56:11.593667 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:11.593642 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc5329cd-398f-42a3-8e4d-82d3b6446773-workload-socket\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jqtlj\" (UID: \"fc5329cd-398f-42a3-8e4d-82d3b6446773\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jqtlj" Oct 30 21:56:11.593791 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:11.593718 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc5329cd-398f-42a3-8e4d-82d3b6446773-workload-certs\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jqtlj\" (UID: \"fc5329cd-398f-42a3-8e4d-82d3b6446773\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jqtlj" Oct 30 21:56:11.595651 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:11.595623 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc5329cd-398f-42a3-8e4d-82d3b6446773-istio-envoy\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jqtlj\" (UID: \"fc5329cd-398f-42a3-8e4d-82d3b6446773\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jqtlj" Oct 30 21:56:11.596103 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:11.596081 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc5329cd-398f-42a3-8e4d-82d3b6446773-istio-podinfo\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jqtlj\" (UID: \"fc5329cd-398f-42a3-8e4d-82d3b6446773\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jqtlj" Oct 30 21:56:11.608749 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:11.608724 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stws9\" (UniqueName: \"kubernetes.io/projected/fc5329cd-398f-42a3-8e4d-82d3b6446773-kube-api-access-stws9\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jqtlj\" (UID: \"fc5329cd-398f-42a3-8e4d-82d3b6446773\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jqtlj" Oct 30 21:56:11.608835 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:11.608793 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc5329cd-398f-42a3-8e4d-82d3b6446773-istio-token\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jqtlj\" (UID: \"fc5329cd-398f-42a3-8e4d-82d3b6446773\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jqtlj" Oct 30 21:56:11.673326 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:11.673292 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jqtlj" Oct 30 21:56:11.797908 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:11.797873 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7694-73b240"] Oct 30 21:56:11.803156 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:11.803133 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7694-73b240" Oct 30 21:56:11.807847 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:11.807790 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-sfm7d\"" Oct 30 21:56:11.810954 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:11.810929 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7694-73b240"] Oct 30 21:56:11.972017 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:11.971989 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jqtlj"] Oct 30 21:56:11.974060 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:56:11.974034 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc5329cd_398f_42a3_8e4d_82d3b6446773.slice/crio-3cabc6e44e7f2e2f41bdea1abfd3483ef86fd0686a683c850084257f7ad79f7f WatchSource:0}: Error finding container 3cabc6e44e7f2e2f41bdea1abfd3483ef86fd0686a683c850084257f7ad79f7f: Status 404 returned error can't find the container with id 3cabc6e44e7f2e2f41bdea1abfd3483ef86fd0686a683c850084257f7ad79f7f Oct 30 21:56:11.975776 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:11.975747 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:11.977817 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:11.977783 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:56:11.977920 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:11.977881 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:56:11.977920 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:11.977914 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:56:11.998922 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:11.998892 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfzlj\" (UniqueName: \"kubernetes.io/projected/014f1489-df6a-4cd7-8b9e-466a2d6fcc51-kube-api-access-hfzlj\") pod \"test-curl-7694-73b240\" (UID: \"014f1489-df6a-4cd7-8b9e-466a2d6fcc51\") " pod="opendatahub/test-curl-7694-73b240" Oct 30 21:56:12.100608 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:12.100573 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfzlj\" (UniqueName: \"kubernetes.io/projected/014f1489-df6a-4cd7-8b9e-466a2d6fcc51-kube-api-access-hfzlj\") pod \"test-curl-7694-73b240\" (UID: \"014f1489-df6a-4cd7-8b9e-466a2d6fcc51\") " pod="opendatahub/test-curl-7694-73b240" Oct 30 21:56:12.113798 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:12.113770 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfzlj\" (UniqueName: \"kubernetes.io/projected/014f1489-df6a-4cd7-8b9e-466a2d6fcc51-kube-api-access-hfzlj\") pod \"test-curl-7694-73b240\" (UID: \"014f1489-df6a-4cd7-8b9e-466a2d6fcc51\") " pod="opendatahub/test-curl-7694-73b240" Oct 30 21:56:12.127601 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:12.127580 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7694-73b240" Oct 30 21:56:12.213747 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:12.212722 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jqtlj" event={"ID":"fc5329cd-398f-42a3-8e4d-82d3b6446773","Type":"ContainerStarted","Data":"2919ba0d0313b3eb597370b898c04f77e02a181debed9e2cd4adc12b32a9fd3e"} Oct 30 21:56:12.213747 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:12.212794 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jqtlj" event={"ID":"fc5329cd-398f-42a3-8e4d-82d3b6446773","Type":"ContainerStarted","Data":"3cabc6e44e7f2e2f41bdea1abfd3483ef86fd0686a683c850084257f7ad79f7f"} Oct 30 21:56:12.240815 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:12.240760 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jqtlj" podStartSLOduration=1.240745408 podStartE2EDuration="1.240745408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:12.23745127 +0000 UTC m=+938.408416450" watchObservedRunningTime="2026-10-30 21:56:12.240745408 +0000 UTC m=+938.411710590" Oct 30 21:56:12.411674 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:12.411647 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7694-73b240"] Oct 30 21:56:12.413048 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:56:12.413019 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod014f1489_df6a_4cd7_8b9e_466a2d6fcc51.slice/crio-9f378261b15131024a69916a145c15e1b9678cfefb31094f30f78ae448de0106 WatchSource:0}: Error finding container 9f378261b15131024a69916a145c15e1b9678cfefb31094f30f78ae448de0106: Status 404 returned error can't find the container with id 9f378261b15131024a69916a145c15e1b9678cfefb31094f30f78ae448de0106 Oct 30 21:56:12.673549 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:12.673478 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jqtlj" Oct 30 21:56:12.788279 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:12.788244 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:13.220284 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:13.220248 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7694-73b240" event={"ID":"014f1489-df6a-4cd7-8b9e-466a2d6fcc51","Type":"ContainerStarted","Data":"9f378261b15131024a69916a145c15e1b9678cfefb31094f30f78ae448de0106"} Oct 30 21:56:13.644030 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:13.643990 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:13.683771 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:13.683266 6720 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jqtlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 21:56:13.683771 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:13.683341 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jqtlj" podUID="fc5329cd-398f-42a3-8e4d-82d3b6446773" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 21:56:14.228299 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:14.228268 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7694-73b240" event={"ID":"014f1489-df6a-4cd7-8b9e-466a2d6fcc51","Type":"ContainerStarted","Data":"b9e8ff162858dbf5fc351401f8b32e6d78137f0b4f2b5bbb1e0635d4fcd58af1"} Oct 30 21:56:14.273050 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:14.273000 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7694-73b240" podStartSLOduration=2.465547582 podStartE2EDuration="3.272979961s" totalImagesPullingTime="807.432379ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:14.269773645 +0000 UTC m=+940.440738828" watchObservedRunningTime="2026-10-30 21:56:14.272979961 +0000 UTC m=+940.443945183" Oct 30 21:56:14.592128 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:14.592042 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:14.673682 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:14.673646 6720 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jqtlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 21:56:14.674178 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:14.673700 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jqtlj" podUID="fc5329cd-398f-42a3-8e4d-82d3b6446773" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 21:56:15.131979 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:15.131928 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7694-73b240"] Oct 30 21:56:15.139083 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:15.139050 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7694-73b240"] Oct 30 21:56:15.427201 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:15.427109 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:15.673857 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:15.673819 6720 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jqtlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 21:56:15.674331 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:15.673896 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jqtlj" podUID="fc5329cd-398f-42a3-8e4d-82d3b6446773" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 21:56:16.241655 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:16.241609 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7694-73b240" podUID="014f1489-df6a-4cd7-8b9e-466a2d6fcc51" containerName="test-curl-7694-73b240" containerID="cri-o://b9e8ff162858dbf5fc351401f8b32e6d78137f0b4f2b5bbb1e0635d4fcd58af1" gracePeriod=2 Oct 30 21:56:16.682196 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:16.682124 6720 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jqtlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 21:56:16.682196 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:16.682184 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jqtlj" podUID="fc5329cd-398f-42a3-8e4d-82d3b6446773" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 21:56:16.683006 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:16.682982 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:17.380776 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:17.380746 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:17.676954 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:17.676865 6720 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jqtlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 21:56:17.677214 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:17.677184 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jqtlj" podUID="fc5329cd-398f-42a3-8e4d-82d3b6446773" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 21:56:18.227241 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:18.227200 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:18.513813 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:18.513782 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7694-73b240" Oct 30 21:56:18.608474 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:18.608444 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/014f1489-df6a-4cd7-8b9e-466a2d6fcc51-kube-api-access-hfzlj\" (UniqueName: \"kubernetes.io/projected/014f1489-df6a-4cd7-8b9e-466a2d6fcc51-kube-api-access-hfzlj\") pod \"014f1489-df6a-4cd7-8b9e-466a2d6fcc51\" (UID: \"014f1489-df6a-4cd7-8b9e-466a2d6fcc51\") " Oct 30 21:56:18.611200 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:18.611177 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/014f1489-df6a-4cd7-8b9e-466a2d6fcc51-kube-api-access-hfzlj" pod "014f1489-df6a-4cd7-8b9e-466a2d6fcc51" (UID: "014f1489-df6a-4cd7-8b9e-466a2d6fcc51"). InnerVolumeSpecName "kube-api-access-hfzlj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:56:18.674530 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:18.673690 6720 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jqtlj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 21:56:18.674530 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:18.673760 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jqtlj" podUID="fc5329cd-398f-42a3-8e4d-82d3b6446773" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 21:56:18.710362 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:18.710331 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfzlj\" (UniqueName: \"kubernetes.io/projected/014f1489-df6a-4cd7-8b9e-466a2d6fcc51-kube-api-access-hfzlj\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:56:19.081293 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:19.081227 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:19.302272 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:19.302195 6720 generic.go:356] "Generic (PLEG): container finished" podID="014f1489-df6a-4cd7-8b9e-466a2d6fcc51" containerID="b9e8ff162858dbf5fc351401f8b32e6d78137f0b4f2b5bbb1e0635d4fcd58af1" exitCode=137 Oct 30 21:56:19.302771 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:19.302313 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7694-73b240" Oct 30 21:56:19.302771 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:19.302343 6720 scope.go:122] "RemoveContainer" containerID="b9e8ff162858dbf5fc351401f8b32e6d78137f0b4f2b5bbb1e0635d4fcd58af1" Oct 30 21:56:19.315059 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:19.315037 6720 scope.go:122] "RemoveContainer" containerID="b9e8ff162858dbf5fc351401f8b32e6d78137f0b4f2b5bbb1e0635d4fcd58af1" Oct 30 21:56:19.315438 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:56:19.315405 6720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9e8ff162858dbf5fc351401f8b32e6d78137f0b4f2b5bbb1e0635d4fcd58af1\": container with ID starting with b9e8ff162858dbf5fc351401f8b32e6d78137f0b4f2b5bbb1e0635d4fcd58af1 not found: ID does not exist" containerID="b9e8ff162858dbf5fc351401f8b32e6d78137f0b4f2b5bbb1e0635d4fcd58af1" Oct 30 21:56:19.315549 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:19.315444 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9e8ff162858dbf5fc351401f8b32e6d78137f0b4f2b5bbb1e0635d4fcd58af1"} err="failed to get container status \"b9e8ff162858dbf5fc351401f8b32e6d78137f0b4f2b5bbb1e0635d4fcd58af1\": rpc error: code = NotFound desc = could not find container \"b9e8ff162858dbf5fc351401f8b32e6d78137f0b4f2b5bbb1e0635d4fcd58af1\": container with ID starting with b9e8ff162858dbf5fc351401f8b32e6d78137f0b4f2b5bbb1e0635d4fcd58af1 not found: ID does not exist" Oct 30 21:56:19.317361 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:56:19.317330 6720 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7694-73b240\" is forbidden: User \"system:node:ip-10-0-137-181.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-181.us-east-2.compute.internal' and this object" podUID="014f1489-df6a-4cd7-8b9e-466a2d6fcc51" pod="opendatahub/test-curl-7694-73b240" Oct 30 21:56:19.962030 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:19.961985 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="014f1489-df6a-4cd7-8b9e-466a2d6fcc51" path="/var/lib/kubelet/pods/014f1489-df6a-4cd7-8b9e-466a2d6fcc51/volumes" Oct 30 21:56:20.037104 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:20.037079 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jqtlj" Oct 30 21:56:20.037522 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:20.037491 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jqtlj" Oct 30 21:56:20.038439 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:20.038420 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jqtlj" Oct 30 21:56:20.178196 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:20.178165 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:20.994009 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:20.993975 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:21.891969 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:21.891935 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:22.611930 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:22.611895 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:23.222013 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:23.221981 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:24.064449 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:24.064406 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:24.508259 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:24.508230 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:25.028262 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:25.028230 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:26.001613 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:26.001585 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:26.400619 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:26.400586 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:26.643979 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:26.643941 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:27.149609 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:27.149565 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:28.055453 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:28.055415 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:28.727192 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:28.727156 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:29.221477 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:29.221447 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:29.403165 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:29.403074 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:30.031663 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:30.031630 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:30.715863 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:30.715791 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:31.585312 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:31.585279 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:32.310945 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:32.310500 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:32.926694 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:32.926645 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:33.728077 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:33.728046 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:34.094659 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:34.094588 6720 scope.go:122] "RemoveContainer" containerID="54006fa841d58f3cc02209ab1e349c816b4104e504b334518de4a990769390b8" Oct 30 21:56:34.103940 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:34.103922 6720 scope.go:122] "RemoveContainer" containerID="e83e55aa77001994ca3eef57ce8b4a6fda551823c925ae0e576b125de4faebee" Oct 30 21:56:34.430510 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:34.430475 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:34.792404 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:34.792316 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:35.130520 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:35.130480 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:35.736142 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:35.736106 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:36.105472 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:36.105443 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:36.314454 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:36.314420 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:36.455318 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:36.455226 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:37.131703 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:37.131664 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:37.836776 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:37.836743 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:38.812934 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:38.812881 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:39.760006 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:39.759969 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-689a2563-7b6b9dcb5f-mnxgq"] Oct 30 21:56:39.760525 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:39.760234 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-689a2563-7b6b9dcb5f-mnxgq" podUID="ca0e6c0a-6b9d-4b0a-b10a-69eaad73dc33" containerName="maas-api" containerID="cri-o://0cc0952e9cdf2848a1e571201ee38967a93d2fb000828b971aae22ec000de204" gracePeriod=30 Oct 30 21:56:39.864947 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:39.864895 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:40.213127 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:40.213094 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-689a2563-7b6b9dcb5f-mnxgq" Oct 30 21:56:40.306980 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:40.306928 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca0e6c0a-6b9d-4b0a-b10a-69eaad73dc33-kube-api-access-9zlsr\" (UniqueName: \"kubernetes.io/projected/ca0e6c0a-6b9d-4b0a-b10a-69eaad73dc33-kube-api-access-9zlsr\") pod \"ca0e6c0a-6b9d-4b0a-b10a-69eaad73dc33\" (UID: \"ca0e6c0a-6b9d-4b0a-b10a-69eaad73dc33\") " Oct 30 21:56:40.309264 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:40.306991 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca0e6c0a-6b9d-4b0a-b10a-69eaad73dc33-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca0e6c0a-6b9d-4b0a-b10a-69eaad73dc33-maas-api-tls\") pod \"ca0e6c0a-6b9d-4b0a-b10a-69eaad73dc33\" (UID: \"ca0e6c0a-6b9d-4b0a-b10a-69eaad73dc33\") " Oct 30 21:56:40.309264 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:40.307029 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca0e6c0a-6b9d-4b0a-b10a-69eaad73dc33-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca0e6c0a-6b9d-4b0a-b10a-69eaad73dc33-openshift-service-ca\") pod \"ca0e6c0a-6b9d-4b0a-b10a-69eaad73dc33\" (UID: \"ca0e6c0a-6b9d-4b0a-b10a-69eaad73dc33\") " Oct 30 21:56:40.322099 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:40.322044 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca0e6c0a-6b9d-4b0a-b10a-69eaad73dc33-openshift-service-ca" pod "ca0e6c0a-6b9d-4b0a-b10a-69eaad73dc33" (UID: "ca0e6c0a-6b9d-4b0a-b10a-69eaad73dc33"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:56:40.322501 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:40.322482 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca0e6c0a-6b9d-4b0a-b10a-69eaad73dc33-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca0e6c0a-6b9d-4b0a-b10a-69eaad73dc33-metrics-certs\") pod \"ca0e6c0a-6b9d-4b0a-b10a-69eaad73dc33\" (UID: \"ca0e6c0a-6b9d-4b0a-b10a-69eaad73dc33\") " Oct 30 21:56:40.324004 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:40.323862 6720 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca0e6c0a-6b9d-4b0a-b10a-69eaad73dc33-openshift-service-ca\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:56:40.328570 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:40.328507 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:40.331794 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:40.331765 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca0e6c0a-6b9d-4b0a-b10a-69eaad73dc33-metrics-certs" pod "ca0e6c0a-6b9d-4b0a-b10a-69eaad73dc33" (UID: "ca0e6c0a-6b9d-4b0a-b10a-69eaad73dc33"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:56:40.332067 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:40.332042 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca0e6c0a-6b9d-4b0a-b10a-69eaad73dc33-kube-api-access-9zlsr" pod "ca0e6c0a-6b9d-4b0a-b10a-69eaad73dc33" (UID: "ca0e6c0a-6b9d-4b0a-b10a-69eaad73dc33"). InnerVolumeSpecName "kube-api-access-9zlsr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:56:40.342875 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:40.339582 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca0e6c0a-6b9d-4b0a-b10a-69eaad73dc33-maas-api-tls" pod "ca0e6c0a-6b9d-4b0a-b10a-69eaad73dc33" (UID: "ca0e6c0a-6b9d-4b0a-b10a-69eaad73dc33"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:56:40.405641 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:56:40.405603 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 21:56:40.425837 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:40.425760 6720 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca0e6c0a-6b9d-4b0a-b10a-69eaad73dc33-metrics-certs\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:56:40.425837 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:40.425790 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zlsr\" (UniqueName: \"kubernetes.io/projected/ca0e6c0a-6b9d-4b0a-b10a-69eaad73dc33-kube-api-access-9zlsr\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:56:40.425837 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:40.425806 6720 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca0e6c0a-6b9d-4b0a-b10a-69eaad73dc33-maas-api-tls\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:56:40.437935 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:40.437901 6720 generic.go:356] "Generic (PLEG): container finished" podID="ca0e6c0a-6b9d-4b0a-b10a-69eaad73dc33" containerID="0cc0952e9cdf2848a1e571201ee38967a93d2fb000828b971aae22ec000de204" exitCode=0 Oct 30 21:56:40.438086 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:40.437958 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-689a2563-7b6b9dcb5f-mnxgq" Oct 30 21:56:40.438086 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:40.437986 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-689a2563-7b6b9dcb5f-mnxgq" event={"ID":"ca0e6c0a-6b9d-4b0a-b10a-69eaad73dc33","Type":"ContainerDied","Data":"0cc0952e9cdf2848a1e571201ee38967a93d2fb000828b971aae22ec000de204"} Oct 30 21:56:40.438086 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:40.438046 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-689a2563-7b6b9dcb5f-mnxgq" event={"ID":"ca0e6c0a-6b9d-4b0a-b10a-69eaad73dc33","Type":"ContainerDied","Data":"87537dd0336458ce1b93a893f2ee21a0ede85548d98525ff0e527e5ce3b4a577"} Oct 30 21:56:40.438086 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:40.438070 6720 scope.go:122] "RemoveContainer" containerID="0cc0952e9cdf2848a1e571201ee38967a93d2fb000828b971aae22ec000de204" Oct 30 21:56:40.449803 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:40.449782 6720 scope.go:122] "RemoveContainer" containerID="0cc0952e9cdf2848a1e571201ee38967a93d2fb000828b971aae22ec000de204" Oct 30 21:56:40.450122 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:56:40.450102 6720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cc0952e9cdf2848a1e571201ee38967a93d2fb000828b971aae22ec000de204\": container with ID starting with 0cc0952e9cdf2848a1e571201ee38967a93d2fb000828b971aae22ec000de204 not found: ID does not exist" containerID="0cc0952e9cdf2848a1e571201ee38967a93d2fb000828b971aae22ec000de204" Oct 30 21:56:40.450198 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:40.450137 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cc0952e9cdf2848a1e571201ee38967a93d2fb000828b971aae22ec000de204"} err="failed to get container status \"0cc0952e9cdf2848a1e571201ee38967a93d2fb000828b971aae22ec000de204\": rpc error: code = NotFound desc = could not find container \"0cc0952e9cdf2848a1e571201ee38967a93d2fb000828b971aae22ec000de204\": container with ID starting with 0cc0952e9cdf2848a1e571201ee38967a93d2fb000828b971aae22ec000de204 not found: ID does not exist" Oct 30 21:56:40.481329 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:40.481300 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-689a2563-7b6b9dcb5f-mnxgq"] Oct 30 21:56:40.486368 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:40.486340 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-689a2563-7b6b9dcb5f-mnxgq"] Oct 30 21:56:40.548296 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:40.548263 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:41.576729 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:41.576683 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:41.957186 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:41.957154 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca0e6c0a-6b9d-4b0a-b10a-69eaad73dc33" path="/var/lib/kubelet/pods/ca0e6c0a-6b9d-4b0a-b10a-69eaad73dc33/volumes" Oct 30 21:56:42.280555 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:42.280470 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:42.571012 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:42.570930 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:42.812028 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:42.811994 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:42.995228 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:42.995193 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:43.278548 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:43.278468 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:43.445647 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:43.445614 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:44.277861 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:44.277830 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:44.772776 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:44.772741 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:45.654526 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:45.654495 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-2r7gp"] Oct 30 21:56:45.655170 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:45.655152 6720 cpu_manager.go:414] "containerMap: removing container" podUID="ca0e6c0a-6b9d-4b0a-b10a-69eaad73dc33" containerName="maas-api" Oct 30 21:56:45.655256 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:45.655172 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca0e6c0a-6b9d-4b0a-b10a-69eaad73dc33" containerName="maas-api" Oct 30 21:56:45.655256 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:45.655201 6720 cpu_manager.go:414] "containerMap: removing container" podUID="014f1489-df6a-4cd7-8b9e-466a2d6fcc51" containerName="test-curl-7694-73b240" Oct 30 21:56:45.655256 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:45.655206 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="014f1489-df6a-4cd7-8b9e-466a2d6fcc51" containerName="test-curl-7694-73b240" Oct 30 21:56:45.655472 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:45.655323 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca0e6c0a-6b9d-4b0a-b10a-69eaad73dc33" containerName="maas-api" Oct 30 21:56:45.655472 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:45.655333 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="014f1489-df6a-4cd7-8b9e-466a2d6fcc51" containerName="test-curl-7694-73b240" Oct 30 21:56:45.660756 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:45.660708 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-2r7gp" Oct 30 21:56:45.694418 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:45.693930 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/895b80be-3b28-4048-a031-09469464216f-workload-socket\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-2r7gp\" (UID: \"895b80be-3b28-4048-a031-09469464216f\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-2r7gp" Oct 30 21:56:45.694418 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:45.694033 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/895b80be-3b28-4048-a031-09469464216f-istio-data\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-2r7gp\" (UID: \"895b80be-3b28-4048-a031-09469464216f\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-2r7gp" Oct 30 21:56:45.694418 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:45.694098 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/895b80be-3b28-4048-a031-09469464216f-istio-token\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-2r7gp\" (UID: \"895b80be-3b28-4048-a031-09469464216f\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-2r7gp" Oct 30 21:56:45.694418 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:45.694121 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/895b80be-3b28-4048-a031-09469464216f-istiod-ca-cert\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-2r7gp\" (UID: \"895b80be-3b28-4048-a031-09469464216f\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-2r7gp" Oct 30 21:56:45.694418 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:45.694224 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/895b80be-3b28-4048-a031-09469464216f-istio-envoy\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-2r7gp\" (UID: \"895b80be-3b28-4048-a031-09469464216f\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-2r7gp" Oct 30 21:56:45.694418 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:45.694284 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/895b80be-3b28-4048-a031-09469464216f-credential-socket\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-2r7gp\" (UID: \"895b80be-3b28-4048-a031-09469464216f\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-2r7gp" Oct 30 21:56:45.694418 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:45.694336 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhdsh\" (UniqueName: \"kubernetes.io/projected/895b80be-3b28-4048-a031-09469464216f-kube-api-access-hhdsh\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-2r7gp\" (UID: \"895b80be-3b28-4048-a031-09469464216f\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-2r7gp" Oct 30 21:56:45.694418 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:45.694363 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/895b80be-3b28-4048-a031-09469464216f-workload-certs\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-2r7gp\" (UID: \"895b80be-3b28-4048-a031-09469464216f\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-2r7gp" Oct 30 21:56:45.694418 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:45.694407 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/895b80be-3b28-4048-a031-09469464216f-istio-podinfo\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-2r7gp\" (UID: \"895b80be-3b28-4048-a031-09469464216f\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-2r7gp" Oct 30 21:56:45.707312 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:45.702611 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-2r7gp"] Oct 30 21:56:45.735845 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:45.735810 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-2c9c95-openshift-default-dockercfg-whdd6\"" Oct 30 21:56:45.742031 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:45.742007 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:45.809322 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:45.809291 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/895b80be-3b28-4048-a031-09469464216f-credential-socket\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-2r7gp\" (UID: \"895b80be-3b28-4048-a031-09469464216f\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-2r7gp" Oct 30 21:56:45.809536 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:45.809355 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhdsh\" (UniqueName: \"kubernetes.io/projected/895b80be-3b28-4048-a031-09469464216f-kube-api-access-hhdsh\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-2r7gp\" (UID: \"895b80be-3b28-4048-a031-09469464216f\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-2r7gp" Oct 30 21:56:45.809536 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:45.809515 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/895b80be-3b28-4048-a031-09469464216f-workload-certs\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-2r7gp\" (UID: \"895b80be-3b28-4048-a031-09469464216f\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-2r7gp" Oct 30 21:56:45.809658 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:45.809555 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/895b80be-3b28-4048-a031-09469464216f-istio-podinfo\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-2r7gp\" (UID: \"895b80be-3b28-4048-a031-09469464216f\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-2r7gp" Oct 30 21:56:45.809719 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:45.809699 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/895b80be-3b28-4048-a031-09469464216f-workload-socket\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-2r7gp\" (UID: \"895b80be-3b28-4048-a031-09469464216f\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-2r7gp" Oct 30 21:56:45.809774 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:45.809730 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/895b80be-3b28-4048-a031-09469464216f-credential-socket\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-2r7gp\" (UID: \"895b80be-3b28-4048-a031-09469464216f\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-2r7gp" Oct 30 21:56:45.809875 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:45.809796 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/895b80be-3b28-4048-a031-09469464216f-istio-data\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-2r7gp\" (UID: \"895b80be-3b28-4048-a031-09469464216f\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-2r7gp" Oct 30 21:56:45.809875 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:45.809850 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/895b80be-3b28-4048-a031-09469464216f-istio-token\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-2r7gp\" (UID: \"895b80be-3b28-4048-a031-09469464216f\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-2r7gp" Oct 30 21:56:45.809875 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:45.809870 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/895b80be-3b28-4048-a031-09469464216f-istiod-ca-cert\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-2r7gp\" (UID: \"895b80be-3b28-4048-a031-09469464216f\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-2r7gp" Oct 30 21:56:45.810052 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:45.809984 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/895b80be-3b28-4048-a031-09469464216f-istio-envoy\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-2r7gp\" (UID: \"895b80be-3b28-4048-a031-09469464216f\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-2r7gp" Oct 30 21:56:45.810107 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:45.810059 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/895b80be-3b28-4048-a031-09469464216f-istio-data\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-2r7gp\" (UID: \"895b80be-3b28-4048-a031-09469464216f\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-2r7gp" Oct 30 21:56:45.810107 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:45.810057 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/895b80be-3b28-4048-a031-09469464216f-workload-socket\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-2r7gp\" (UID: \"895b80be-3b28-4048-a031-09469464216f\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-2r7gp" Oct 30 21:56:45.810281 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:45.809869 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/895b80be-3b28-4048-a031-09469464216f-workload-certs\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-2r7gp\" (UID: \"895b80be-3b28-4048-a031-09469464216f\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-2r7gp" Oct 30 21:56:45.810786 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:45.810611 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/895b80be-3b28-4048-a031-09469464216f-istiod-ca-cert\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-2r7gp\" (UID: \"895b80be-3b28-4048-a031-09469464216f\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-2r7gp" Oct 30 21:56:45.812686 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:45.812655 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/895b80be-3b28-4048-a031-09469464216f-istio-envoy\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-2r7gp\" (UID: \"895b80be-3b28-4048-a031-09469464216f\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-2r7gp" Oct 30 21:56:45.813035 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:45.813012 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/895b80be-3b28-4048-a031-09469464216f-istio-podinfo\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-2r7gp\" (UID: \"895b80be-3b28-4048-a031-09469464216f\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-2r7gp" Oct 30 21:56:45.825535 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:45.825507 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/895b80be-3b28-4048-a031-09469464216f-istio-token\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-2r7gp\" (UID: \"895b80be-3b28-4048-a031-09469464216f\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-2r7gp" Oct 30 21:56:45.825650 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:45.825599 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhdsh\" (UniqueName: \"kubernetes.io/projected/895b80be-3b28-4048-a031-09469464216f-kube-api-access-hhdsh\") pod \"e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-2r7gp\" (UID: \"895b80be-3b28-4048-a031-09469464216f\") " pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-2r7gp" Oct 30 21:56:45.975856 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:45.975776 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-2r7gp" Oct 30 21:56:46.068457 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:46.061915 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:46.246682 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:46.246606 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:46.298106 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:46.298080 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-2r7gp"] Oct 30 21:56:46.304836 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:56:46.304806 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod895b80be_3b28_4048_a031_09469464216f.slice/crio-605bbe0d34c081401ca5d7321b5496a7bed6916a3f51c40436b8d3ecefcc3ddd WatchSource:0}: Error finding container 605bbe0d34c081401ca5d7321b5496a7bed6916a3f51c40436b8d3ecefcc3ddd: Status 404 returned error can't find the container with id 605bbe0d34c081401ca5d7321b5496a7bed6916a3f51c40436b8d3ecefcc3ddd Oct 30 21:56:46.307455 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:46.307418 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:56:46.307559 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:46.307511 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:56:46.307559 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:46.307550 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:56:46.483792 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:46.483742 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-2r7gp" event={"ID":"895b80be-3b28-4048-a031-09469464216f","Type":"ContainerStarted","Data":"1e8e8004c9e5c9e49eea0451cbc335e85f0e33980c1b8ecd458cdd3eb9d05236"} Oct 30 21:56:46.483993 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:46.483828 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-2r7gp" event={"ID":"895b80be-3b28-4048-a031-09469464216f","Type":"ContainerStarted","Data":"605bbe0d34c081401ca5d7321b5496a7bed6916a3f51c40436b8d3ecefcc3ddd"} Oct 30 21:56:46.511892 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:46.511778 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-2r7gp" podStartSLOduration=1.51176036 podStartE2EDuration="1.51176036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:46.507406463 +0000 UTC m=+972.678371648" watchObservedRunningTime="2026-10-30 21:56:46.51176036 +0000 UTC m=+972.682725543" Oct 30 21:56:46.897172 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:46.897141 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:46.976851 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:46.976816 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-2r7gp" Oct 30 21:56:47.410608 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:47.408986 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:47.599344 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:47.599308 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:47.979290 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:47.979257 6720 patch_prober.go:29] interesting pod/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-2r7gp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 21:56:47.979742 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:47.979311 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-2r7gp" podUID="895b80be-3b28-4048-a031-09469464216f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 21:56:48.291251 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:48.290047 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:48.945925 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:48.945889 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:48.976969 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:48.976934 6720 patch_prober.go:29] interesting pod/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-2r7gp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 21:56:48.977236 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:48.977206 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-2r7gp" podUID="895b80be-3b28-4048-a031-09469464216f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 21:56:49.684874 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:49.684848 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2c9c95-55d58b4c57-hk5bq"] Oct 30 21:56:49.685524 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:49.685494 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c9c95-55d58b4c57-hk5bq" podUID="43a8441e-d4cc-4802-a621-ffdbe0e13e0b" containerName="maas-api" containerID="cri-o://87bad815139a4f47f51b85a63a700b1d03d7e5c15192bf71332a22ecdf52c79a" gracePeriod=30 Oct 30 21:56:49.781471 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:49.778793 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:49.987627 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:49.977103 6720 patch_prober.go:29] interesting pod/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-2r7gp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 21:56:49.987627 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:49.977163 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-2r7gp" podUID="895b80be-3b28-4048-a031-09469464216f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 21:56:50.123927 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:50.122662 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:50.236238 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:50.236207 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c9c95-55d58b4c57-hk5bq" Oct 30 21:56:50.308709 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:50.308558 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:50.334482 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:50.334454 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43a8441e-d4cc-4802-a621-ffdbe0e13e0b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43a8441e-d4cc-4802-a621-ffdbe0e13e0b-maas-api-tls\") pod \"43a8441e-d4cc-4802-a621-ffdbe0e13e0b\" (UID: \"43a8441e-d4cc-4802-a621-ffdbe0e13e0b\") " Oct 30 21:56:50.334656 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:50.334630 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43a8441e-d4cc-4802-a621-ffdbe0e13e0b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/43a8441e-d4cc-4802-a621-ffdbe0e13e0b-metrics-certs\") pod \"43a8441e-d4cc-4802-a621-ffdbe0e13e0b\" (UID: \"43a8441e-d4cc-4802-a621-ffdbe0e13e0b\") " Oct 30 21:56:50.334656 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:50.334649 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43a8441e-d4cc-4802-a621-ffdbe0e13e0b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43a8441e-d4cc-4802-a621-ffdbe0e13e0b-openshift-service-ca\") pod \"43a8441e-d4cc-4802-a621-ffdbe0e13e0b\" (UID: \"43a8441e-d4cc-4802-a621-ffdbe0e13e0b\") " Oct 30 21:56:50.334780 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:50.334708 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43a8441e-d4cc-4802-a621-ffdbe0e13e0b-kube-api-access-5njxf\" (UniqueName: \"kubernetes.io/projected/43a8441e-d4cc-4802-a621-ffdbe0e13e0b-kube-api-access-5njxf\") pod \"43a8441e-d4cc-4802-a621-ffdbe0e13e0b\" (UID: \"43a8441e-d4cc-4802-a621-ffdbe0e13e0b\") " Oct 30 21:56:50.335001 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:50.334978 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43a8441e-d4cc-4802-a621-ffdbe0e13e0b-openshift-service-ca" pod "43a8441e-d4cc-4802-a621-ffdbe0e13e0b" (UID: "43a8441e-d4cc-4802-a621-ffdbe0e13e0b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:56:50.335486 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:50.335466 6720 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43a8441e-d4cc-4802-a621-ffdbe0e13e0b-openshift-service-ca\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:56:50.336802 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:50.336776 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43a8441e-d4cc-4802-a621-ffdbe0e13e0b-maas-api-tls" pod "43a8441e-d4cc-4802-a621-ffdbe0e13e0b" (UID: "43a8441e-d4cc-4802-a621-ffdbe0e13e0b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:56:50.336895 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:50.336812 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43a8441e-d4cc-4802-a621-ffdbe0e13e0b-kube-api-access-5njxf" pod "43a8441e-d4cc-4802-a621-ffdbe0e13e0b" (UID: "43a8441e-d4cc-4802-a621-ffdbe0e13e0b"). InnerVolumeSpecName "kube-api-access-5njxf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:56:50.336895 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:50.336827 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43a8441e-d4cc-4802-a621-ffdbe0e13e0b-metrics-certs" pod "43a8441e-d4cc-4802-a621-ffdbe0e13e0b" (UID: "43a8441e-d4cc-4802-a621-ffdbe0e13e0b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:56:50.438749 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:50.437804 6720 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43a8441e-d4cc-4802-a621-ffdbe0e13e0b-metrics-certs\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:56:50.438749 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:50.437843 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5njxf\" (UniqueName: \"kubernetes.io/projected/43a8441e-d4cc-4802-a621-ffdbe0e13e0b-kube-api-access-5njxf\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:56:50.438749 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:50.437862 6720 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43a8441e-d4cc-4802-a621-ffdbe0e13e0b-maas-api-tls\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:56:50.485063 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:50.485033 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:50.523301 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:50.523266 6720 generic.go:356] "Generic (PLEG): container finished" podID="43a8441e-d4cc-4802-a621-ffdbe0e13e0b" containerID="87bad815139a4f47f51b85a63a700b1d03d7e5c15192bf71332a22ecdf52c79a" exitCode=0 Oct 30 21:56:50.523488 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:50.523319 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c9c95-55d58b4c57-hk5bq" Oct 30 21:56:50.523488 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:50.523344 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c9c95-55d58b4c57-hk5bq" event={"ID":"43a8441e-d4cc-4802-a621-ffdbe0e13e0b","Type":"ContainerDied","Data":"87bad815139a4f47f51b85a63a700b1d03d7e5c15192bf71332a22ecdf52c79a"} Oct 30 21:56:50.523488 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:50.523418 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-2c9c95-55d58b4c57-hk5bq" event={"ID":"43a8441e-d4cc-4802-a621-ffdbe0e13e0b","Type":"ContainerDied","Data":"f072dff4a331295e06015e43c8df1d81221607721324781436d78f353f13aba4"} Oct 30 21:56:50.523488 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:50.523445 6720 scope.go:122] "RemoveContainer" containerID="87bad815139a4f47f51b85a63a700b1d03d7e5c15192bf71332a22ecdf52c79a" Oct 30 21:56:50.534601 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:50.534583 6720 scope.go:122] "RemoveContainer" containerID="87bad815139a4f47f51b85a63a700b1d03d7e5c15192bf71332a22ecdf52c79a" Oct 30 21:56:50.534890 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:56:50.534871 6720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87bad815139a4f47f51b85a63a700b1d03d7e5c15192bf71332a22ecdf52c79a\": container with ID starting with 87bad815139a4f47f51b85a63a700b1d03d7e5c15192bf71332a22ecdf52c79a not found: ID does not exist" containerID="87bad815139a4f47f51b85a63a700b1d03d7e5c15192bf71332a22ecdf52c79a" Oct 30 21:56:50.534939 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:50.534900 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87bad815139a4f47f51b85a63a700b1d03d7e5c15192bf71332a22ecdf52c79a"} err="failed to get container status \"87bad815139a4f47f51b85a63a700b1d03d7e5c15192bf71332a22ecdf52c79a\": rpc error: code = NotFound desc = could not find container \"87bad815139a4f47f51b85a63a700b1d03d7e5c15192bf71332a22ecdf52c79a\": container with ID starting with 87bad815139a4f47f51b85a63a700b1d03d7e5c15192bf71332a22ecdf52c79a not found: ID does not exist" Oct 30 21:56:50.581470 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:50.581036 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2c9c95-55d58b4c57-hk5bq"] Oct 30 21:56:50.583729 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:50.583700 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-2c9c95-55d58b4c57-hk5bq"] Oct 30 21:56:50.818544 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:50.818507 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:50.977364 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:50.977330 6720 patch_prober.go:29] interesting pod/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-2r7gp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 21:56:50.977557 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:50.977406 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-2r7gp" podUID="895b80be-3b28-4048-a031-09469464216f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 21:56:51.305979 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:51.296052 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:51.977366 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:51.977336 6720 patch_prober.go:29] interesting pod/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-2r7gp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 21:56:51.977797 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:51.977412 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-2r7gp" podUID="895b80be-3b28-4048-a031-09469464216f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 21:56:51.980646 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:51.980617 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43a8441e-d4cc-4802-a621-ffdbe0e13e0b" path="/var/lib/kubelet/pods/43a8441e-d4cc-4802-a621-ffdbe0e13e0b/volumes" Oct 30 21:56:52.416817 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:52.416786 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:52.733538 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:52.733456 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:52.896031 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:52.896000 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:53.192986 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:53.192954 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:53.248490 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:53.248456 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-03f0a0-865f88d5c8-smxql"] Oct 30 21:56:53.249711 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:53.249612 6720 cpu_manager.go:414] "containerMap: removing container" podUID="43a8441e-d4cc-4802-a621-ffdbe0e13e0b" containerName="maas-api" Oct 30 21:56:53.249884 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:53.249864 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43a8441e-d4cc-4802-a621-ffdbe0e13e0b" containerName="maas-api" Oct 30 21:56:53.250669 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:53.250650 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="43a8441e-d4cc-4802-a621-ffdbe0e13e0b" containerName="maas-api" Oct 30 21:56:53.259479 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:53.259460 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-03f0a0-865f88d5c8-smxql" Oct 30 21:56:53.268820 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:53.268799 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-03f0a0-865f88d5c8-smxql"] Oct 30 21:56:53.294311 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:53.293846 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-2r7gp" Oct 30 21:56:53.295529 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:53.295473 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-2r7gp" Oct 30 21:56:53.297311 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:53.297292 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-2r7gp" Oct 30 21:56:53.322052 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:53.322013 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-03f0a0-serving-cert\"" Oct 30 21:56:53.418315 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:53.418281 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a924070-ecb2-4f0a-94f2-a280a470bd55-metrics-certs\") pod \"maas-api-e2e-api-a-03f0a0-865f88d5c8-smxql\" (UID: \"1a924070-ecb2-4f0a-94f2-a280a470bd55\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-03f0a0-865f88d5c8-smxql" Oct 30 21:56:53.418511 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:53.418442 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a924070-ecb2-4f0a-94f2-a280a470bd55-openshift-service-ca\") pod \"maas-api-e2e-api-a-03f0a0-865f88d5c8-smxql\" (UID: \"1a924070-ecb2-4f0a-94f2-a280a470bd55\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-03f0a0-865f88d5c8-smxql" Oct 30 21:56:53.418511 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:53.418500 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p64ws\" (UniqueName: \"kubernetes.io/projected/1a924070-ecb2-4f0a-94f2-a280a470bd55-kube-api-access-p64ws\") pod \"maas-api-e2e-api-a-03f0a0-865f88d5c8-smxql\" (UID: \"1a924070-ecb2-4f0a-94f2-a280a470bd55\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-03f0a0-865f88d5c8-smxql" Oct 30 21:56:53.418643 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:53.418610 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a924070-ecb2-4f0a-94f2-a280a470bd55-maas-api-tls\") pod \"maas-api-e2e-api-a-03f0a0-865f88d5c8-smxql\" (UID: \"1a924070-ecb2-4f0a-94f2-a280a470bd55\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-03f0a0-865f88d5c8-smxql" Oct 30 21:56:53.469081 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:53.469000 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:53.520161 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:53.520131 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a924070-ecb2-4f0a-94f2-a280a470bd55-openshift-service-ca\") pod \"maas-api-e2e-api-a-03f0a0-865f88d5c8-smxql\" (UID: \"1a924070-ecb2-4f0a-94f2-a280a470bd55\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-03f0a0-865f88d5c8-smxql" Oct 30 21:56:53.520436 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:53.520186 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p64ws\" (UniqueName: \"kubernetes.io/projected/1a924070-ecb2-4f0a-94f2-a280a470bd55-kube-api-access-p64ws\") pod \"maas-api-e2e-api-a-03f0a0-865f88d5c8-smxql\" (UID: \"1a924070-ecb2-4f0a-94f2-a280a470bd55\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-03f0a0-865f88d5c8-smxql" Oct 30 21:56:53.520436 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:53.520235 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a924070-ecb2-4f0a-94f2-a280a470bd55-maas-api-tls\") pod \"maas-api-e2e-api-a-03f0a0-865f88d5c8-smxql\" (UID: \"1a924070-ecb2-4f0a-94f2-a280a470bd55\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-03f0a0-865f88d5c8-smxql" Oct 30 21:56:53.520583 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:53.520445 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a924070-ecb2-4f0a-94f2-a280a470bd55-metrics-certs\") pod \"maas-api-e2e-api-a-03f0a0-865f88d5c8-smxql\" (UID: \"1a924070-ecb2-4f0a-94f2-a280a470bd55\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-03f0a0-865f88d5c8-smxql" Oct 30 21:56:53.520899 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:53.520877 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a924070-ecb2-4f0a-94f2-a280a470bd55-openshift-service-ca\") pod \"maas-api-e2e-api-a-03f0a0-865f88d5c8-smxql\" (UID: \"1a924070-ecb2-4f0a-94f2-a280a470bd55\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-03f0a0-865f88d5c8-smxql" Oct 30 21:56:53.523039 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:53.523020 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a924070-ecb2-4f0a-94f2-a280a470bd55-metrics-certs\") pod \"maas-api-e2e-api-a-03f0a0-865f88d5c8-smxql\" (UID: \"1a924070-ecb2-4f0a-94f2-a280a470bd55\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-03f0a0-865f88d5c8-smxql" Oct 30 21:56:53.523105 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:53.523020 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a924070-ecb2-4f0a-94f2-a280a470bd55-maas-api-tls\") pod \"maas-api-e2e-api-a-03f0a0-865f88d5c8-smxql\" (UID: \"1a924070-ecb2-4f0a-94f2-a280a470bd55\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-03f0a0-865f88d5c8-smxql" Oct 30 21:56:53.536353 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:53.536326 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p64ws\" (UniqueName: \"kubernetes.io/projected/1a924070-ecb2-4f0a-94f2-a280a470bd55-kube-api-access-p64ws\") pod \"maas-api-e2e-api-a-03f0a0-865f88d5c8-smxql\" (UID: \"1a924070-ecb2-4f0a-94f2-a280a470bd55\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-03f0a0-865f88d5c8-smxql" Oct 30 21:56:53.606458 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:53.589659 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-03f0a0-865f88d5c8-smxql" Oct 30 21:56:53.715030 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:53.714994 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:53.924782 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:53.924743 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-03f0a0-865f88d5c8-smxql"] Oct 30 21:56:54.329444 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:54.329366 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:54.557763 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:54.557732 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-03f0a0-865f88d5c8-smxql" event={"ID":"1a924070-ecb2-4f0a-94f2-a280a470bd55","Type":"ContainerStarted","Data":"d3f186789464b138e3459e564eb7cef4cc1e0156afe1c861604e0d918b29dc04"} Oct 30 21:56:54.557983 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:54.557787 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-03f0a0-865f88d5c8-smxql" event={"ID":"1a924070-ecb2-4f0a-94f2-a280a470bd55","Type":"ContainerStarted","Data":"a131d727d02e8efba588ea03e19562b1dc50cb30947dcf6c0b262aaf7cba5254"} Oct 30 21:56:54.557983 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:54.557952 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-03f0a0-865f88d5c8-smxql" Oct 30 21:56:54.582556 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:54.582475 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-03f0a0-865f88d5c8-smxql" podStartSLOduration=1.200997032 podStartE2EDuration="1.582461306s" totalImagesPullingTime="381.464274ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:54.579143742 +0000 UTC m=+980.750108934" watchObservedRunningTime="2026-10-30 21:56:54.582461306 +0000 UTC m=+980.753426488" Oct 30 21:56:55.239971 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:55.239925 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:56.354407 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:56.353903 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:57.322420 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:57.321247 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:57.697944 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:57.697897 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:57.871959 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:57.871924 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:58.020579 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:58.019084 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-2r7gp"] Oct 30 21:56:58.025417 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:58.022527 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-2r7gp" podUID="895b80be-3b28-4048-a031-09469464216f" containerName="istio-proxy" containerID="cri-o://1e8e8004c9e5c9e49eea0451cbc335e85f0e33980c1b8ecd458cdd3eb9d05236" gracePeriod=30 Oct 30 21:56:58.227164 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:58.226959 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:58.410017 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:58.409976 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:59.307664 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:59.307630 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:56:59.926054 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:59.926019 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-689a2563-7b6b9dcb5f-hf5mp"] Oct 30 21:56:59.931003 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:59.930980 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-689a2563-7b6b9dcb5f-hf5mp" Oct 30 21:56:59.946960 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:59.946929 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-689a2563-serving-cert\"" Oct 30 21:56:59.961101 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:56:59.961071 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-689a2563-7b6b9dcb5f-hf5mp"] Oct 30 21:57:00.023952 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:00.023920 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:00.058295 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:00.058250 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcncq\" (UniqueName: \"kubernetes.io/projected/7e7456b2-3efc-42fe-9c0d-9889d89ca562-kube-api-access-gcncq\") pod \"maas-api-e2e-mt-689a2563-7b6b9dcb5f-hf5mp\" (UID: \"7e7456b2-3efc-42fe-9c0d-9889d89ca562\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-689a2563-7b6b9dcb5f-hf5mp" Oct 30 21:57:00.058544 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:00.058366 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e7456b2-3efc-42fe-9c0d-9889d89ca562-maas-api-tls\") pod \"maas-api-e2e-mt-689a2563-7b6b9dcb5f-hf5mp\" (UID: \"7e7456b2-3efc-42fe-9c0d-9889d89ca562\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-689a2563-7b6b9dcb5f-hf5mp" Oct 30 21:57:00.058544 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:00.058438 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e7456b2-3efc-42fe-9c0d-9889d89ca562-metrics-certs\") pod \"maas-api-e2e-mt-689a2563-7b6b9dcb5f-hf5mp\" (UID: \"7e7456b2-3efc-42fe-9c0d-9889d89ca562\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-689a2563-7b6b9dcb5f-hf5mp" Oct 30 21:57:00.058544 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:00.058471 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e7456b2-3efc-42fe-9c0d-9889d89ca562-openshift-service-ca\") pod \"maas-api-e2e-mt-689a2563-7b6b9dcb5f-hf5mp\" (UID: \"7e7456b2-3efc-42fe-9c0d-9889d89ca562\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-689a2563-7b6b9dcb5f-hf5mp" Oct 30 21:57:00.159651 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:00.159616 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e7456b2-3efc-42fe-9c0d-9889d89ca562-metrics-certs\") pod \"maas-api-e2e-mt-689a2563-7b6b9dcb5f-hf5mp\" (UID: \"7e7456b2-3efc-42fe-9c0d-9889d89ca562\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-689a2563-7b6b9dcb5f-hf5mp" Oct 30 21:57:00.159834 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:00.159686 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e7456b2-3efc-42fe-9c0d-9889d89ca562-openshift-service-ca\") pod \"maas-api-e2e-mt-689a2563-7b6b9dcb5f-hf5mp\" (UID: \"7e7456b2-3efc-42fe-9c0d-9889d89ca562\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-689a2563-7b6b9dcb5f-hf5mp" Oct 30 21:57:00.159906 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:00.159887 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcncq\" (UniqueName: \"kubernetes.io/projected/7e7456b2-3efc-42fe-9c0d-9889d89ca562-kube-api-access-gcncq\") pod \"maas-api-e2e-mt-689a2563-7b6b9dcb5f-hf5mp\" (UID: \"7e7456b2-3efc-42fe-9c0d-9889d89ca562\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-689a2563-7b6b9dcb5f-hf5mp" Oct 30 21:57:00.160008 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:00.159991 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e7456b2-3efc-42fe-9c0d-9889d89ca562-maas-api-tls\") pod \"maas-api-e2e-mt-689a2563-7b6b9dcb5f-hf5mp\" (UID: \"7e7456b2-3efc-42fe-9c0d-9889d89ca562\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-689a2563-7b6b9dcb5f-hf5mp" Oct 30 21:57:00.160159 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:57:00.160137 6720 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-689a2563-serving-cert: secret "maas-api-e2e-mt-689a2563-serving-cert" not found Oct 30 21:57:00.160243 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:57:00.160232 6720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e7456b2-3efc-42fe-9c0d-9889d89ca562-maas-api-tls podName:7e7456b2-3efc-42fe-9c0d-9889d89ca562 nodeName:}" failed. No retries permitted until 2026-10-30 21:57:00.660211126 +0000 UTC m=+986.831176285 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7e7456b2-3efc-42fe-9c0d-9889d89ca562-maas-api-tls") pod "maas-api-e2e-mt-689a2563-7b6b9dcb5f-hf5mp" (UID: "7e7456b2-3efc-42fe-9c0d-9889d89ca562") : secret "maas-api-e2e-mt-689a2563-serving-cert" not found Oct 30 21:57:00.160544 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:00.160518 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e7456b2-3efc-42fe-9c0d-9889d89ca562-openshift-service-ca\") pod \"maas-api-e2e-mt-689a2563-7b6b9dcb5f-hf5mp\" (UID: \"7e7456b2-3efc-42fe-9c0d-9889d89ca562\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-689a2563-7b6b9dcb5f-hf5mp" Oct 30 21:57:00.162684 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:00.162663 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e7456b2-3efc-42fe-9c0d-9889d89ca562-metrics-certs\") pod \"maas-api-e2e-mt-689a2563-7b6b9dcb5f-hf5mp\" (UID: \"7e7456b2-3efc-42fe-9c0d-9889d89ca562\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-689a2563-7b6b9dcb5f-hf5mp" Oct 30 21:57:00.175346 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:00.175305 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcncq\" (UniqueName: \"kubernetes.io/projected/7e7456b2-3efc-42fe-9c0d-9889d89ca562-kube-api-access-gcncq\") pod \"maas-api-e2e-mt-689a2563-7b6b9dcb5f-hf5mp\" (UID: \"7e7456b2-3efc-42fe-9c0d-9889d89ca562\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-689a2563-7b6b9dcb5f-hf5mp" Oct 30 21:57:00.196626 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:00.196551 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:00.366049 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:00.366020 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:00.542326 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:00.542151 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:00.574123 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:00.574087 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-03f0a0-865f88d5c8-smxql" Oct 30 21:57:00.675287 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:00.675254 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e7456b2-3efc-42fe-9c0d-9889d89ca562-maas-api-tls\") pod \"maas-api-e2e-mt-689a2563-7b6b9dcb5f-hf5mp\" (UID: \"7e7456b2-3efc-42fe-9c0d-9889d89ca562\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-689a2563-7b6b9dcb5f-hf5mp" Oct 30 21:57:00.680152 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:00.680122 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e7456b2-3efc-42fe-9c0d-9889d89ca562-maas-api-tls\") pod \"maas-api-e2e-mt-689a2563-7b6b9dcb5f-hf5mp\" (UID: \"7e7456b2-3efc-42fe-9c0d-9889d89ca562\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-689a2563-7b6b9dcb5f-hf5mp" Oct 30 21:57:00.855938 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:00.855907 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-689a2563-7b6b9dcb5f-hf5mp" Oct 30 21:57:00.950215 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:00.950149 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:01.259598 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:01.259566 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-689a2563-7b6b9dcb5f-hf5mp"] Oct 30 21:57:01.262624 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:57:01.262594 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e7456b2_3efc_42fe_9c0d_9889d89ca562.slice/crio-c1190f746fc7a217b81f680397cc0f50f7a455671d94006cae53e1b8ddc5943c WatchSource:0}: Error finding container c1190f746fc7a217b81f680397cc0f50f7a455671d94006cae53e1b8ddc5943c: Status 404 returned error can't find the container with id c1190f746fc7a217b81f680397cc0f50f7a455671d94006cae53e1b8ddc5943c Oct 30 21:57:01.274202 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:01.274182 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:01.713596 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:01.713566 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-689a2563-7b6b9dcb5f-hf5mp" event={"ID":"7e7456b2-3efc-42fe-9c0d-9889d89ca562","Type":"ContainerStarted","Data":"c1190f746fc7a217b81f680397cc0f50f7a455671d94006cae53e1b8ddc5943c"} Oct 30 21:57:01.944336 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:01.944229 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:02.723050 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:02.723011 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-689a2563-7b6b9dcb5f-hf5mp" event={"ID":"7e7456b2-3efc-42fe-9c0d-9889d89ca562","Type":"ContainerStarted","Data":"1a1688f46f2c16935ef0081ba2049a595d47c48a9d0b358d0015af9b4dd6c24d"} Oct 30 21:57:02.723542 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:02.723256 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-689a2563-7b6b9dcb5f-hf5mp" Oct 30 21:57:02.747049 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:02.746994 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-689a2563-7b6b9dcb5f-hf5mp" podStartSLOduration=3.322393992 podStartE2EDuration="3.746977786s" totalImagesPullingTime="424.583794ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:02.745545599 +0000 UTC m=+988.916510818" watchObservedRunningTime="2026-10-30 21:57:02.746977786 +0000 UTC m=+988.917942975" Oct 30 21:57:02.790509 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:02.790472 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:03.354858 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:03.354829 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-2r7gp" Oct 30 21:57:03.428307 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:03.428269 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/895b80be-3b28-4048-a031-09469464216f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/895b80be-3b28-4048-a031-09469464216f-workload-certs\") pod \"895b80be-3b28-4048-a031-09469464216f\" (UID: \"895b80be-3b28-4048-a031-09469464216f\") " Oct 30 21:57:03.428495 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:03.428462 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/895b80be-3b28-4048-a031-09469464216f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/895b80be-3b28-4048-a031-09469464216f-istiod-ca-cert\") pod \"895b80be-3b28-4048-a031-09469464216f\" (UID: \"895b80be-3b28-4048-a031-09469464216f\") " Oct 30 21:57:03.428565 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:03.428547 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/895b80be-3b28-4048-a031-09469464216f-istio-token\" (UniqueName: \"kubernetes.io/projected/895b80be-3b28-4048-a031-09469464216f-istio-token\") pod \"895b80be-3b28-4048-a031-09469464216f\" (UID: \"895b80be-3b28-4048-a031-09469464216f\") " Oct 30 21:57:03.428624 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:03.428576 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/895b80be-3b28-4048-a031-09469464216f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/895b80be-3b28-4048-a031-09469464216f-credential-socket\") pod \"895b80be-3b28-4048-a031-09469464216f\" (UID: \"895b80be-3b28-4048-a031-09469464216f\") " Oct 30 21:57:03.428624 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:03.428597 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/895b80be-3b28-4048-a031-09469464216f-workload-certs" pod "895b80be-3b28-4048-a031-09469464216f" (UID: "895b80be-3b28-4048-a031-09469464216f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:03.428741 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:03.428620 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/895b80be-3b28-4048-a031-09469464216f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/895b80be-3b28-4048-a031-09469464216f-istio-envoy\") pod \"895b80be-3b28-4048-a031-09469464216f\" (UID: \"895b80be-3b28-4048-a031-09469464216f\") " Oct 30 21:57:03.428741 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:03.428661 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/895b80be-3b28-4048-a031-09469464216f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/895b80be-3b28-4048-a031-09469464216f-istio-podinfo\") pod \"895b80be-3b28-4048-a031-09469464216f\" (UID: \"895b80be-3b28-4048-a031-09469464216f\") " Oct 30 21:57:03.428741 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:03.428704 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/895b80be-3b28-4048-a031-09469464216f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/895b80be-3b28-4048-a031-09469464216f-workload-socket\") pod \"895b80be-3b28-4048-a031-09469464216f\" (UID: \"895b80be-3b28-4048-a031-09469464216f\") " Oct 30 21:57:03.428896 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:03.428782 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/895b80be-3b28-4048-a031-09469464216f-kube-api-access-hhdsh\" (UniqueName: \"kubernetes.io/projected/895b80be-3b28-4048-a031-09469464216f-kube-api-access-hhdsh\") pod \"895b80be-3b28-4048-a031-09469464216f\" (UID: \"895b80be-3b28-4048-a031-09469464216f\") " Oct 30 21:57:03.428896 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:03.428827 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/895b80be-3b28-4048-a031-09469464216f-credential-socket" pod "895b80be-3b28-4048-a031-09469464216f" (UID: "895b80be-3b28-4048-a031-09469464216f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:03.428896 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:03.428866 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/895b80be-3b28-4048-a031-09469464216f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/895b80be-3b28-4048-a031-09469464216f-istio-data\") pod \"895b80be-3b28-4048-a031-09469464216f\" (UID: \"895b80be-3b28-4048-a031-09469464216f\") " Oct 30 21:57:03.428896 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:03.428867 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/895b80be-3b28-4048-a031-09469464216f-istiod-ca-cert" pod "895b80be-3b28-4048-a031-09469464216f" (UID: "895b80be-3b28-4048-a031-09469464216f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:57:03.429136 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:03.429114 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/895b80be-3b28-4048-a031-09469464216f-istio-data" pod "895b80be-3b28-4048-a031-09469464216f" (UID: "895b80be-3b28-4048-a031-09469464216f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:03.429603 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:03.429576 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/895b80be-3b28-4048-a031-09469464216f-workload-socket" pod "895b80be-3b28-4048-a031-09469464216f" (UID: "895b80be-3b28-4048-a031-09469464216f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:03.429746 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:03.429671 6720 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/895b80be-3b28-4048-a031-09469464216f-istio-data\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:03.429867 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:03.429844 6720 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/895b80be-3b28-4048-a031-09469464216f-workload-certs\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:03.429962 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:03.429873 6720 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/895b80be-3b28-4048-a031-09469464216f-istiod-ca-cert\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:03.429962 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:03.429888 6720 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/895b80be-3b28-4048-a031-09469464216f-credential-socket\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:03.431988 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:03.431963 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/895b80be-3b28-4048-a031-09469464216f-istio-token" pod "895b80be-3b28-4048-a031-09469464216f" (UID: "895b80be-3b28-4048-a031-09469464216f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:03.432076 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:03.431963 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/895b80be-3b28-4048-a031-09469464216f-istio-podinfo" pod "895b80be-3b28-4048-a031-09469464216f" (UID: "895b80be-3b28-4048-a031-09469464216f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:57:03.432076 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:03.432031 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/895b80be-3b28-4048-a031-09469464216f-kube-api-access-hhdsh" pod "895b80be-3b28-4048-a031-09469464216f" (UID: "895b80be-3b28-4048-a031-09469464216f"). InnerVolumeSpecName "kube-api-access-hhdsh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:03.435558 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:03.435533 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/895b80be-3b28-4048-a031-09469464216f-istio-envoy" pod "895b80be-3b28-4048-a031-09469464216f" (UID: "895b80be-3b28-4048-a031-09469464216f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:03.531474 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:03.531426 6720 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/895b80be-3b28-4048-a031-09469464216f-istio-token\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:03.531474 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:03.531462 6720 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/895b80be-3b28-4048-a031-09469464216f-istio-envoy\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:03.531474 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:03.531476 6720 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/895b80be-3b28-4048-a031-09469464216f-istio-podinfo\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:03.531980 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:03.531491 6720 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/895b80be-3b28-4048-a031-09469464216f-workload-socket\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:03.531980 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:03.531508 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhdsh\" (UniqueName: \"kubernetes.io/projected/895b80be-3b28-4048-a031-09469464216f-kube-api-access-hhdsh\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:03.532466 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:03.532435 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:03.607070 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:03.607037 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k5rml"] Oct 30 21:57:03.607699 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:03.607674 6720 cpu_manager.go:414] "containerMap: removing container" podUID="895b80be-3b28-4048-a031-09469464216f" containerName="istio-proxy" Oct 30 21:57:03.607819 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:03.607701 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="895b80be-3b28-4048-a031-09469464216f" containerName="istio-proxy" Oct 30 21:57:03.607878 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:03.607866 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="895b80be-3b28-4048-a031-09469464216f" containerName="istio-proxy" Oct 30 21:57:03.612322 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:03.612288 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k5rml" Oct 30 21:57:03.623404 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:03.623352 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 21:57:03.623674 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:03.623647 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 21:57:03.629890 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:03.627011 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k5rml"] Oct 30 21:57:03.629890 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:03.627684 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-lx85x\"" Oct 30 21:57:03.629890 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:03.627721 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 21:57:03.632675 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:03.632647 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fa558cd-48e6-425c-a2cc-f4a75d24a20f-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k5rml\" (UID: \"2fa558cd-48e6-425c-a2cc-f4a75d24a20f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k5rml" Oct 30 21:57:03.632820 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:03.632691 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2fa558cd-48e6-425c-a2cc-f4a75d24a20f-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k5rml\" (UID: \"2fa558cd-48e6-425c-a2cc-f4a75d24a20f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k5rml" Oct 30 21:57:03.632820 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:03.632783 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2fa558cd-48e6-425c-a2cc-f4a75d24a20f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k5rml\" (UID: \"2fa558cd-48e6-425c-a2cc-f4a75d24a20f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k5rml" Oct 30 21:57:03.632936 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:03.632886 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2fa558cd-48e6-425c-a2cc-f4a75d24a20f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k5rml\" (UID: \"2fa558cd-48e6-425c-a2cc-f4a75d24a20f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k5rml" Oct 30 21:57:03.632992 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:03.632944 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fa558cd-48e6-425c-a2cc-f4a75d24a20f-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k5rml\" (UID: \"2fa558cd-48e6-425c-a2cc-f4a75d24a20f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k5rml" Oct 30 21:57:03.633040 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:03.633009 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2fa558cd-48e6-425c-a2cc-f4a75d24a20f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k5rml\" (UID: \"2fa558cd-48e6-425c-a2cc-f4a75d24a20f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k5rml" Oct 30 21:57:03.633236 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:03.633213 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bscf\" (UniqueName: \"kubernetes.io/projected/2fa558cd-48e6-425c-a2cc-f4a75d24a20f-kube-api-access-9bscf\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k5rml\" (UID: \"2fa558cd-48e6-425c-a2cc-f4a75d24a20f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k5rml" Oct 30 21:57:03.730115 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:03.730082 6720 generic.go:356] "Generic (PLEG): container finished" podID="895b80be-3b28-4048-a031-09469464216f" containerID="1e8e8004c9e5c9e49eea0451cbc335e85f0e33980c1b8ecd458cdd3eb9d05236" exitCode=0 Oct 30 21:57:03.730573 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:03.730132 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-2r7gp" Oct 30 21:57:03.730573 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:03.730163 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-2r7gp" event={"ID":"895b80be-3b28-4048-a031-09469464216f","Type":"ContainerDied","Data":"1e8e8004c9e5c9e49eea0451cbc335e85f0e33980c1b8ecd458cdd3eb9d05236"} Oct 30 21:57:03.730573 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:03.730214 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-2r7gp" event={"ID":"895b80be-3b28-4048-a031-09469464216f","Type":"ContainerDied","Data":"605bbe0d34c081401ca5d7321b5496a7bed6916a3f51c40436b8d3ecefcc3ddd"} Oct 30 21:57:03.730573 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:03.730245 6720 scope.go:122] "RemoveContainer" containerID="1e8e8004c9e5c9e49eea0451cbc335e85f0e33980c1b8ecd458cdd3eb9d05236" Oct 30 21:57:03.734340 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:03.734316 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bscf\" (UniqueName: \"kubernetes.io/projected/2fa558cd-48e6-425c-a2cc-f4a75d24a20f-kube-api-access-9bscf\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k5rml\" (UID: \"2fa558cd-48e6-425c-a2cc-f4a75d24a20f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k5rml" Oct 30 21:57:03.734499 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:03.734417 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fa558cd-48e6-425c-a2cc-f4a75d24a20f-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k5rml\" (UID: \"2fa558cd-48e6-425c-a2cc-f4a75d24a20f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k5rml" Oct 30 21:57:03.734600 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:03.734572 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2fa558cd-48e6-425c-a2cc-f4a75d24a20f-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k5rml\" (UID: \"2fa558cd-48e6-425c-a2cc-f4a75d24a20f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k5rml" Oct 30 21:57:03.734739 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:03.734618 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2fa558cd-48e6-425c-a2cc-f4a75d24a20f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k5rml\" (UID: \"2fa558cd-48e6-425c-a2cc-f4a75d24a20f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k5rml" Oct 30 21:57:03.734739 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:03.734687 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2fa558cd-48e6-425c-a2cc-f4a75d24a20f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k5rml\" (UID: \"2fa558cd-48e6-425c-a2cc-f4a75d24a20f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k5rml" Oct 30 21:57:03.734870 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:03.734765 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fa558cd-48e6-425c-a2cc-f4a75d24a20f-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k5rml\" (UID: \"2fa558cd-48e6-425c-a2cc-f4a75d24a20f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k5rml" Oct 30 21:57:03.734870 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:03.734796 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2fa558cd-48e6-425c-a2cc-f4a75d24a20f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k5rml\" (UID: \"2fa558cd-48e6-425c-a2cc-f4a75d24a20f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k5rml" Oct 30 21:57:03.735463 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:03.735442 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2fa558cd-48e6-425c-a2cc-f4a75d24a20f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k5rml\" (UID: \"2fa558cd-48e6-425c-a2cc-f4a75d24a20f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k5rml" Oct 30 21:57:03.737586 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:03.737534 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fa558cd-48e6-425c-a2cc-f4a75d24a20f-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k5rml\" (UID: \"2fa558cd-48e6-425c-a2cc-f4a75d24a20f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k5rml" Oct 30 21:57:03.737772 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:03.737753 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2fa558cd-48e6-425c-a2cc-f4a75d24a20f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k5rml\" (UID: \"2fa558cd-48e6-425c-a2cc-f4a75d24a20f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k5rml" Oct 30 21:57:03.737850 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:03.737818 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2fa558cd-48e6-425c-a2cc-f4a75d24a20f-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k5rml\" (UID: \"2fa558cd-48e6-425c-a2cc-f4a75d24a20f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k5rml" Oct 30 21:57:03.737897 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:03.737869 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2fa558cd-48e6-425c-a2cc-f4a75d24a20f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k5rml\" (UID: \"2fa558cd-48e6-425c-a2cc-f4a75d24a20f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k5rml" Oct 30 21:57:03.742006 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:03.741989 6720 scope.go:122] "RemoveContainer" containerID="1e8e8004c9e5c9e49eea0451cbc335e85f0e33980c1b8ecd458cdd3eb9d05236" Oct 30 21:57:03.742294 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:57:03.742277 6720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e8e8004c9e5c9e49eea0451cbc335e85f0e33980c1b8ecd458cdd3eb9d05236\": container with ID starting with 1e8e8004c9e5c9e49eea0451cbc335e85f0e33980c1b8ecd458cdd3eb9d05236 not found: ID does not exist" containerID="1e8e8004c9e5c9e49eea0451cbc335e85f0e33980c1b8ecd458cdd3eb9d05236" Oct 30 21:57:03.742339 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:03.742302 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e8e8004c9e5c9e49eea0451cbc335e85f0e33980c1b8ecd458cdd3eb9d05236"} err="failed to get container status \"1e8e8004c9e5c9e49eea0451cbc335e85f0e33980c1b8ecd458cdd3eb9d05236\": rpc error: code = NotFound desc = could not find container \"1e8e8004c9e5c9e49eea0451cbc335e85f0e33980c1b8ecd458cdd3eb9d05236\": container with ID starting with 1e8e8004c9e5c9e49eea0451cbc335e85f0e33980c1b8ecd458cdd3eb9d05236 not found: ID does not exist" Oct 30 21:57:03.749190 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:03.749164 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fa558cd-48e6-425c-a2cc-f4a75d24a20f-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k5rml\" (UID: \"2fa558cd-48e6-425c-a2cc-f4a75d24a20f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k5rml" Oct 30 21:57:03.750293 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:03.750274 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bscf\" (UniqueName: \"kubernetes.io/projected/2fa558cd-48e6-425c-a2cc-f4a75d24a20f-kube-api-access-9bscf\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k5rml\" (UID: \"2fa558cd-48e6-425c-a2cc-f4a75d24a20f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k5rml" Oct 30 21:57:03.762866 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:03.762840 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-2r7gp"] Oct 30 21:57:03.770889 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:03.770867 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-2c9c95-openshift-default-5f8bd69f4c-2r7gp"] Oct 30 21:57:03.931041 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:03.930958 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k5rml" Oct 30 21:57:03.957674 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:03.957647 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="895b80be-3b28-4048-a031-09469464216f" path="/var/lib/kubelet/pods/895b80be-3b28-4048-a031-09469464216f/volumes" Oct 30 21:57:04.224162 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:04.224135 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k5rml"] Oct 30 21:57:04.225120 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:57:04.225096 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fa558cd_48e6_425c_a2cc_f4a75d24a20f.slice/crio-52a29f2340f696ef40512a7099c6bc61f1845592711b98331f82a4fa14877f46 WatchSource:0}: Error finding container 52a29f2340f696ef40512a7099c6bc61f1845592711b98331f82a4fa14877f46: Status 404 returned error can't find the container with id 52a29f2340f696ef40512a7099c6bc61f1845592711b98331f82a4fa14877f46 Oct 30 21:57:04.485571 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:04.485485 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:04.738067 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:04.737986 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k5rml" event={"ID":"2fa558cd-48e6-425c-a2cc-f4a75d24a20f","Type":"ContainerStarted","Data":"52a29f2340f696ef40512a7099c6bc61f1845592711b98331f82a4fa14877f46"} Oct 30 21:57:05.050608 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:05.050502 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:05.227809 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:05.227777 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:05.457134 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:05.457101 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:06.032703 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:06.032672 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:06.321169 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:06.321045 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:06.493013 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:06.492723 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:06.686362 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:06.686331 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:07.284962 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:07.284893 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:07.450303 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:07.450269 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:07.462462 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:07.462409 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:57:07.462625 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:07.462509 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:57:07.608410 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:07.607517 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:07.765800 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:07.765759 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k5rml" event={"ID":"2fa558cd-48e6-425c-a2cc-f4a75d24a20f","Type":"ContainerStarted","Data":"09a94182d9eea04ca8720c9c623359521cd3f0cc2862820568dd33747bbf77c5"} Oct 30 21:57:07.766008 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:07.765844 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k5rml" Oct 30 21:57:07.793651 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:07.793595 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k5rml" podStartSLOduration=1.560545208 podStartE2EDuration="4.793578303s" totalImagesPullingTime="3.233033095s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:07.791055831 +0000 UTC m=+993.962021036" watchObservedRunningTime="2026-10-30 21:57:07.793578303 +0000 UTC m=+993.964543486" Oct 30 21:57:08.231287 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:08.228358 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:08.737437 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:08.737365 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-689a2563-7b6b9dcb5f-hf5mp" Oct 30 21:57:08.778192 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:08.778148 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k5rml" Oct 30 21:57:08.913720 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:08.913688 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:09.063971 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:09.063861 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:09.227142 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:09.227105 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:09.694869 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:09.694840 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:10.403849 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:57:10.403806 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 21:57:10.431933 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:10.431907 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:10.822735 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:10.822633 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:11.021018 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:11.020992 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:11.280977 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:11.280937 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:11.484084 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:11.484037 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:12.316171 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:12.316141 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:12.580334 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:12.580238 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:13.715889 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:13.715787 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:14.570123 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:14.570088 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:15.339949 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:15.339911 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:16.168153 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:16.168123 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:17.079436 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:17.079379 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:18.210014 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:18.209973 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:18.443410 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:18.443219 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:19.604479 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:19.599848 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/premium-w3-ceb36a-kserve-6576bb476f-j5kdr" podUID="d065e788-c972-4683-84ed-86a70ed4791d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:57:19.618923 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:19.618790 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:19.795024 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:19.793760 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/distinct-w4-939a16-kserve-7f88f68c56-cngtk" podUID="85316c69-7ebd-4fb6-9195-44d1fba2dfbc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 21:57:19.915499 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:19.915353 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/distinct-2-w1-0f26c5-kserve-74976f985c-fw28x" podUID="97e1e65d-1d50-4097-98ca-2944fa182807" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:57:19.946572 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:19.946526 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-689a2563/e2e-lifecycle-model-689a2563-kserve-5bb4c5b87c-q6gl8" podUID="450c5ec5-0a77-449d-a515-1aa21776a572" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:57:20.621924 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:20.621883 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:21.446993 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:21.446946 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:22.360992 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:22.360951 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:22.514299 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:22.514258 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:23.217923 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:23.215912 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:23.911139 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:23.911111 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:24.417457 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:24.417417 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:24.568984 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:24.568953 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:24.718531 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:24.718453 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:25.156849 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:25.156814 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:25.332849 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:25.332816 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:25.597543 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:25.597446 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:25.927141 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:25.927102 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:26.459374 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:26.459287 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:26.731118 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:26.731029 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:26.910255 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:26.910222 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:27.106237 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:27.106206 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:27.621928 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:27.621891 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:28.708921 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:28.708885 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:29.641898 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:29.641869 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:30.595475 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:30.595440 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:30.801819 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:30.801791 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:31.021555 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:31.021525 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:31.718935 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:31.718856 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:31.922287 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:31.922258 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:32.757010 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:32.756980 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:33.739263 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:33.739229 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:34.440610 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:34.440525 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:34.617424 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:34.617363 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:34.906407 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:34.902117 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:35.313342 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:35.313263 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:36.033103 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:36.033067 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:36.715837 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:36.715803 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:37.418103 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:37.418040 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:37.634806 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:37.634764 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:38.234646 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:38.234607 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:38.383263 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:38.383222 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:39.119729 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:39.119687 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:39.847946 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:39.847911 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:40.402784 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:57:40.402737 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 21:57:40.550234 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:40.550197 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:41.314914 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:41.314877 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:42.092196 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:42.091798 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:42.820163 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:42.820122 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:43.280492 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:43.280458 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-689a2563-7b6b9dcb5f-hf5mp"] Oct 30 21:57:43.281304 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:43.281254 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-689a2563-7b6b9dcb5f-hf5mp" podUID="7e7456b2-3efc-42fe-9c0d-9889d89ca562" containerName="maas-api" containerID="cri-o://1a1688f46f2c16935ef0081ba2049a595d47c48a9d0b358d0015af9b4dd6c24d" gracePeriod=30 Oct 30 21:57:43.302523 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:43.301620 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:43.499427 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:43.499366 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:43.691443 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:43.689785 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:43.744076 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:43.744047 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-689a2563-7b6b9dcb5f-hf5mp" Oct 30 21:57:43.896362 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:43.896325 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e7456b2-3efc-42fe-9c0d-9889d89ca562-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e7456b2-3efc-42fe-9c0d-9889d89ca562-openshift-service-ca\") pod \"7e7456b2-3efc-42fe-9c0d-9889d89ca562\" (UID: \"7e7456b2-3efc-42fe-9c0d-9889d89ca562\") " Oct 30 21:57:43.896602 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:43.896429 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e7456b2-3efc-42fe-9c0d-9889d89ca562-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e7456b2-3efc-42fe-9c0d-9889d89ca562-metrics-certs\") pod \"7e7456b2-3efc-42fe-9c0d-9889d89ca562\" (UID: \"7e7456b2-3efc-42fe-9c0d-9889d89ca562\") " Oct 30 21:57:43.896677 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:43.896614 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e7456b2-3efc-42fe-9c0d-9889d89ca562-kube-api-access-gcncq\" (UniqueName: \"kubernetes.io/projected/7e7456b2-3efc-42fe-9c0d-9889d89ca562-kube-api-access-gcncq\") pod \"7e7456b2-3efc-42fe-9c0d-9889d89ca562\" (UID: \"7e7456b2-3efc-42fe-9c0d-9889d89ca562\") " Oct 30 21:57:43.896758 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:43.896741 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e7456b2-3efc-42fe-9c0d-9889d89ca562-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e7456b2-3efc-42fe-9c0d-9889d89ca562-maas-api-tls\") pod \"7e7456b2-3efc-42fe-9c0d-9889d89ca562\" (UID: \"7e7456b2-3efc-42fe-9c0d-9889d89ca562\") " Oct 30 21:57:43.896881 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:43.896848 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e7456b2-3efc-42fe-9c0d-9889d89ca562-openshift-service-ca" pod "7e7456b2-3efc-42fe-9c0d-9889d89ca562" (UID: "7e7456b2-3efc-42fe-9c0d-9889d89ca562"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:57:43.897364 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:43.897341 6720 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e7456b2-3efc-42fe-9c0d-9889d89ca562-openshift-service-ca\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:43.899631 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:43.899596 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e7456b2-3efc-42fe-9c0d-9889d89ca562-maas-api-tls" pod "7e7456b2-3efc-42fe-9c0d-9889d89ca562" (UID: "7e7456b2-3efc-42fe-9c0d-9889d89ca562"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:57:43.901044 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:43.900999 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e7456b2-3efc-42fe-9c0d-9889d89ca562-metrics-certs" pod "7e7456b2-3efc-42fe-9c0d-9889d89ca562" (UID: "7e7456b2-3efc-42fe-9c0d-9889d89ca562"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:57:43.901168 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:43.901145 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e7456b2-3efc-42fe-9c0d-9889d89ca562-kube-api-access-gcncq" pod "7e7456b2-3efc-42fe-9c0d-9889d89ca562" (UID: "7e7456b2-3efc-42fe-9c0d-9889d89ca562"). InnerVolumeSpecName "kube-api-access-gcncq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:43.955620 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:43.955539 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:43.999583 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:43.999417 6720 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e7456b2-3efc-42fe-9c0d-9889d89ca562-maas-api-tls\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:43.999583 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:43.999455 6720 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e7456b2-3efc-42fe-9c0d-9889d89ca562-metrics-certs\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:43.999583 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:43.999473 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcncq\" (UniqueName: \"kubernetes.io/projected/7e7456b2-3efc-42fe-9c0d-9889d89ca562-kube-api-access-gcncq\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:44.098087 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:44.098053 6720 generic.go:356] "Generic (PLEG): container finished" podID="7e7456b2-3efc-42fe-9c0d-9889d89ca562" containerID="1a1688f46f2c16935ef0081ba2049a595d47c48a9d0b358d0015af9b4dd6c24d" exitCode=0 Oct 30 21:57:44.098268 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:44.098122 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-689a2563-7b6b9dcb5f-hf5mp" Oct 30 21:57:44.098268 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:44.098133 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-689a2563-7b6b9dcb5f-hf5mp" event={"ID":"7e7456b2-3efc-42fe-9c0d-9889d89ca562","Type":"ContainerDied","Data":"1a1688f46f2c16935ef0081ba2049a595d47c48a9d0b358d0015af9b4dd6c24d"} Oct 30 21:57:44.098268 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:44.098193 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-689a2563-7b6b9dcb5f-hf5mp" event={"ID":"7e7456b2-3efc-42fe-9c0d-9889d89ca562","Type":"ContainerDied","Data":"c1190f746fc7a217b81f680397cc0f50f7a455671d94006cae53e1b8ddc5943c"} Oct 30 21:57:44.098268 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:44.098215 6720 scope.go:122] "RemoveContainer" containerID="1a1688f46f2c16935ef0081ba2049a595d47c48a9d0b358d0015af9b4dd6c24d" Oct 30 21:57:44.111043 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:44.111007 6720 scope.go:122] "RemoveContainer" containerID="1a1688f46f2c16935ef0081ba2049a595d47c48a9d0b358d0015af9b4dd6c24d" Oct 30 21:57:44.111435 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:57:44.111376 6720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a1688f46f2c16935ef0081ba2049a595d47c48a9d0b358d0015af9b4dd6c24d\": container with ID starting with 1a1688f46f2c16935ef0081ba2049a595d47c48a9d0b358d0015af9b4dd6c24d not found: ID does not exist" containerID="1a1688f46f2c16935ef0081ba2049a595d47c48a9d0b358d0015af9b4dd6c24d" Oct 30 21:57:44.111524 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:44.111449 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a1688f46f2c16935ef0081ba2049a595d47c48a9d0b358d0015af9b4dd6c24d"} err="failed to get container status \"1a1688f46f2c16935ef0081ba2049a595d47c48a9d0b358d0015af9b4dd6c24d\": rpc error: code = NotFound desc = could not find container \"1a1688f46f2c16935ef0081ba2049a595d47c48a9d0b358d0015af9b4dd6c24d\": container with ID starting with 1a1688f46f2c16935ef0081ba2049a595d47c48a9d0b358d0015af9b4dd6c24d not found: ID does not exist" Oct 30 21:57:44.121854 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:44.121825 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-689a2563-7b6b9dcb5f-hf5mp"] Oct 30 21:57:44.126760 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:44.126739 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-689a2563-7b6b9dcb5f-hf5mp"] Oct 30 21:57:44.254425 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:44.254300 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:44.740045 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:44.740016 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:45.825334 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:45.825278 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:45.958413 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:45.958369 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e7456b2-3efc-42fe-9c0d-9889d89ca562" path="/var/lib/kubelet/pods/7e7456b2-3efc-42fe-9c0d-9889d89ca562/volumes" Oct 30 21:57:46.744408 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:46.736225 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-689a2563-7b6b9dcb5f-hf5mp" podUID="7e7456b2-3efc-42fe-9c0d-9889d89ca562" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.83:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:57:47.057751 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:47.057622 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:47.926179 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:47.926146 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:48.609000 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:48.605679 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:49.164924 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:49.164893 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-689a2563/e2e-lifecycle-model-689a2563-kserve-5bb4c5b87c-q6gl8"] Oct 30 21:57:49.259432 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:49.259355 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:49.450801 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:49.450681 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:49.825252 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:49.825148 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:50.341217 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:50.341181 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:50.511259 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:50.511223 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:51.113734 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:51.113704 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:52.082969 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:52.082939 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:52.576716 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:52.576684 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-03f0a0-865f88d5c8-smxql"] Oct 30 21:57:52.577179 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:52.576901 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-03f0a0-865f88d5c8-smxql" podUID="1a924070-ecb2-4f0a-94f2-a280a470bd55" containerName="maas-api" containerID="cri-o://d3f186789464b138e3459e564eb7cef4cc1e0156afe1c861604e0d918b29dc04" gracePeriod=30 Oct 30 21:57:52.652113 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:52.652067 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:52.931419 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:52.929637 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:53.030714 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:53.030687 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-03f0a0-865f88d5c8-smxql" Oct 30 21:57:53.092141 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:53.092110 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:53.126087 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:53.126044 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a924070-ecb2-4f0a-94f2-a280a470bd55-kube-api-access-p64ws\" (UniqueName: \"kubernetes.io/projected/1a924070-ecb2-4f0a-94f2-a280a470bd55-kube-api-access-p64ws\") pod \"1a924070-ecb2-4f0a-94f2-a280a470bd55\" (UID: \"1a924070-ecb2-4f0a-94f2-a280a470bd55\") " Oct 30 21:57:53.126256 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:53.126148 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a924070-ecb2-4f0a-94f2-a280a470bd55-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a924070-ecb2-4f0a-94f2-a280a470bd55-metrics-certs\") pod \"1a924070-ecb2-4f0a-94f2-a280a470bd55\" (UID: \"1a924070-ecb2-4f0a-94f2-a280a470bd55\") " Oct 30 21:57:53.126324 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:53.126251 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a924070-ecb2-4f0a-94f2-a280a470bd55-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a924070-ecb2-4f0a-94f2-a280a470bd55-maas-api-tls\") pod \"1a924070-ecb2-4f0a-94f2-a280a470bd55\" (UID: \"1a924070-ecb2-4f0a-94f2-a280a470bd55\") " Oct 30 21:57:53.126324 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:53.126312 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a924070-ecb2-4f0a-94f2-a280a470bd55-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a924070-ecb2-4f0a-94f2-a280a470bd55-openshift-service-ca\") pod \"1a924070-ecb2-4f0a-94f2-a280a470bd55\" (UID: \"1a924070-ecb2-4f0a-94f2-a280a470bd55\") " Oct 30 21:57:53.126759 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:53.126733 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a924070-ecb2-4f0a-94f2-a280a470bd55-openshift-service-ca" pod "1a924070-ecb2-4f0a-94f2-a280a470bd55" (UID: "1a924070-ecb2-4f0a-94f2-a280a470bd55"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:57:53.127170 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:53.127140 6720 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1a924070-ecb2-4f0a-94f2-a280a470bd55-openshift-service-ca\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:53.128571 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:53.128543 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a924070-ecb2-4f0a-94f2-a280a470bd55-metrics-certs" pod "1a924070-ecb2-4f0a-94f2-a280a470bd55" (UID: "1a924070-ecb2-4f0a-94f2-a280a470bd55"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:57:53.128952 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:53.128929 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a924070-ecb2-4f0a-94f2-a280a470bd55-maas-api-tls" pod "1a924070-ecb2-4f0a-94f2-a280a470bd55" (UID: "1a924070-ecb2-4f0a-94f2-a280a470bd55"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:57:53.129020 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:53.128957 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a924070-ecb2-4f0a-94f2-a280a470bd55-kube-api-access-p64ws" pod "1a924070-ecb2-4f0a-94f2-a280a470bd55" (UID: "1a924070-ecb2-4f0a-94f2-a280a470bd55"). InnerVolumeSpecName "kube-api-access-p64ws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:53.158659 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:53.158628 6720 generic.go:356] "Generic (PLEG): container finished" podID="1a924070-ecb2-4f0a-94f2-a280a470bd55" containerID="d3f186789464b138e3459e564eb7cef4cc1e0156afe1c861604e0d918b29dc04" exitCode=0 Oct 30 21:57:53.158835 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:53.158688 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-03f0a0-865f88d5c8-smxql" Oct 30 21:57:53.158835 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:53.158710 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-03f0a0-865f88d5c8-smxql" event={"ID":"1a924070-ecb2-4f0a-94f2-a280a470bd55","Type":"ContainerDied","Data":"d3f186789464b138e3459e564eb7cef4cc1e0156afe1c861604e0d918b29dc04"} Oct 30 21:57:53.158835 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:53.158766 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-03f0a0-865f88d5c8-smxql" event={"ID":"1a924070-ecb2-4f0a-94f2-a280a470bd55","Type":"ContainerDied","Data":"a131d727d02e8efba588ea03e19562b1dc50cb30947dcf6c0b262aaf7cba5254"} Oct 30 21:57:53.158835 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:53.158803 6720 scope.go:122] "RemoveContainer" containerID="d3f186789464b138e3459e564eb7cef4cc1e0156afe1c861604e0d918b29dc04" Oct 30 21:57:53.178978 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:53.174246 6720 scope.go:122] "RemoveContainer" containerID="d3f186789464b138e3459e564eb7cef4cc1e0156afe1c861604e0d918b29dc04" Oct 30 21:57:53.178978 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:57:53.175033 6720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3f186789464b138e3459e564eb7cef4cc1e0156afe1c861604e0d918b29dc04\": container with ID starting with d3f186789464b138e3459e564eb7cef4cc1e0156afe1c861604e0d918b29dc04 not found: ID does not exist" containerID="d3f186789464b138e3459e564eb7cef4cc1e0156afe1c861604e0d918b29dc04" Oct 30 21:57:53.178978 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:53.175081 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3f186789464b138e3459e564eb7cef4cc1e0156afe1c861604e0d918b29dc04"} err="failed to get container status \"d3f186789464b138e3459e564eb7cef4cc1e0156afe1c861604e0d918b29dc04\": rpc error: code = NotFound desc = could not find container \"d3f186789464b138e3459e564eb7cef4cc1e0156afe1c861604e0d918b29dc04\": container with ID starting with d3f186789464b138e3459e564eb7cef4cc1e0156afe1c861604e0d918b29dc04 not found: ID does not exist" Oct 30 21:57:53.190707 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:53.190623 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-03f0a0-865f88d5c8-smxql"] Oct 30 21:57:53.195153 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:53.195104 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-03f0a0-865f88d5c8-smxql"] Oct 30 21:57:53.228470 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:53.228435 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p64ws\" (UniqueName: \"kubernetes.io/projected/1a924070-ecb2-4f0a-94f2-a280a470bd55-kube-api-access-p64ws\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:53.228470 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:53.228472 6720 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a924070-ecb2-4f0a-94f2-a280a470bd55-metrics-certs\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:53.228685 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:53.228490 6720 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1a924070-ecb2-4f0a-94f2-a280a470bd55-maas-api-tls\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:53.253489 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:53.253448 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:53.587219 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:53.587098 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:53.879131 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:53.879089 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:53.961513 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:53.961478 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a924070-ecb2-4f0a-94f2-a280a470bd55" path="/var/lib/kubelet/pods/1a924070-ecb2-4f0a-94f2-a280a470bd55/volumes" Oct 30 21:57:54.816479 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:54.816440 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:55.651698 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:55.651671 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:56.424736 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:56.424703 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:57.022767 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:57.022732 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:57.757112 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:57.757078 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:58.425644 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:58.425604 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:59.146836 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:59.146808 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:57:59.909691 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:57:59.909655 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:00.330469 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:00.330354 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:00.550187 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:00.550153 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:00.736089 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:00.736034 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:01.794903 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:01.794848 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:02.002278 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:02.002188 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:02.909816 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:02.909778 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:03.713651 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:03.713616 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:04.188716 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:04.188654 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-689a2563/e2e-lifecycle-model-689a2563-kserve-5bb4c5b87c-q6gl8" podUID="450c5ec5-0a77-449d-a515-1aa21776a572" containerName="main" containerID="cri-o://a2280742c1687520374c40f9e9eb2b7a0374bddacb7a65d6fef0d5b788a23851" gracePeriod=45 Oct 30 21:58:04.434903 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:04.434864 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:04.475519 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:04.475489 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-689a2563/e2e-lifecycle-model-689a2563-kserve-5bb4c5b87c-q6gl8" Oct 30 21:58:04.585813 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:04.585781 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/450c5ec5-0a77-449d-a515-1aa21776a572-dshm\" (UniqueName: \"kubernetes.io/empty-dir/450c5ec5-0a77-449d-a515-1aa21776a572-dshm\") pod \"450c5ec5-0a77-449d-a515-1aa21776a572\" (UID: \"450c5ec5-0a77-449d-a515-1aa21776a572\") " Oct 30 21:58:04.585997 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:04.585961 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/450c5ec5-0a77-449d-a515-1aa21776a572-tls-certs\" (UniqueName: \"kubernetes.io/secret/450c5ec5-0a77-449d-a515-1aa21776a572-tls-certs\") pod \"450c5ec5-0a77-449d-a515-1aa21776a572\" (UID: \"450c5ec5-0a77-449d-a515-1aa21776a572\") " Oct 30 21:58:04.586062 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:04.585998 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/450c5ec5-0a77-449d-a515-1aa21776a572-kube-api-access-vvq7n\" (UniqueName: \"kubernetes.io/projected/450c5ec5-0a77-449d-a515-1aa21776a572-kube-api-access-vvq7n\") pod \"450c5ec5-0a77-449d-a515-1aa21776a572\" (UID: \"450c5ec5-0a77-449d-a515-1aa21776a572\") " Oct 30 21:58:04.586128 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:04.586081 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/450c5ec5-0a77-449d-a515-1aa21776a572-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/450c5ec5-0a77-449d-a515-1aa21776a572-tmp-dir\") pod \"450c5ec5-0a77-449d-a515-1aa21776a572\" (UID: \"450c5ec5-0a77-449d-a515-1aa21776a572\") " Oct 30 21:58:04.586128 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:04.586112 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/450c5ec5-0a77-449d-a515-1aa21776a572-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/450c5ec5-0a77-449d-a515-1aa21776a572-model-cache\") pod \"450c5ec5-0a77-449d-a515-1aa21776a572\" (UID: \"450c5ec5-0a77-449d-a515-1aa21776a572\") " Oct 30 21:58:04.586231 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:04.586163 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/450c5ec5-0a77-449d-a515-1aa21776a572-home\" (UniqueName: \"kubernetes.io/empty-dir/450c5ec5-0a77-449d-a515-1aa21776a572-home\") pod \"450c5ec5-0a77-449d-a515-1aa21776a572\" (UID: \"450c5ec5-0a77-449d-a515-1aa21776a572\") " Oct 30 21:58:04.586427 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:04.586377 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/450c5ec5-0a77-449d-a515-1aa21776a572-tmp-dir" pod "450c5ec5-0a77-449d-a515-1aa21776a572" (UID: "450c5ec5-0a77-449d-a515-1aa21776a572"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:58:04.586518 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:04.586435 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/450c5ec5-0a77-449d-a515-1aa21776a572-model-cache" pod "450c5ec5-0a77-449d-a515-1aa21776a572" (UID: "450c5ec5-0a77-449d-a515-1aa21776a572"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:58:04.586577 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:04.586551 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/450c5ec5-0a77-449d-a515-1aa21776a572-home" pod "450c5ec5-0a77-449d-a515-1aa21776a572" (UID: "450c5ec5-0a77-449d-a515-1aa21776a572"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:58:04.587023 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:04.586995 6720 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/450c5ec5-0a77-449d-a515-1aa21776a572-tmp-dir\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:04.587146 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:04.587025 6720 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/450c5ec5-0a77-449d-a515-1aa21776a572-model-cache\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:04.587146 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:04.587041 6720 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/450c5ec5-0a77-449d-a515-1aa21776a572-home\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:04.588492 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:04.588464 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/450c5ec5-0a77-449d-a515-1aa21776a572-tls-certs" pod "450c5ec5-0a77-449d-a515-1aa21776a572" (UID: "450c5ec5-0a77-449d-a515-1aa21776a572"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:58:04.588769 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:04.588746 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/450c5ec5-0a77-449d-a515-1aa21776a572-dshm" pod "450c5ec5-0a77-449d-a515-1aa21776a572" (UID: "450c5ec5-0a77-449d-a515-1aa21776a572"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:58:04.590795 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:04.590761 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/450c5ec5-0a77-449d-a515-1aa21776a572-kube-api-access-vvq7n" pod "450c5ec5-0a77-449d-a515-1aa21776a572" (UID: "450c5ec5-0a77-449d-a515-1aa21776a572"). InnerVolumeSpecName "kube-api-access-vvq7n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:58:04.688108 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:04.688076 6720 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/450c5ec5-0a77-449d-a515-1aa21776a572-dshm\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:04.688108 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:04.688105 6720 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/450c5ec5-0a77-449d-a515-1aa21776a572-tls-certs\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:04.688319 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:04.688116 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvq7n\" (UniqueName: \"kubernetes.io/projected/450c5ec5-0a77-449d-a515-1aa21776a572-kube-api-access-vvq7n\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:58:05.171212 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:05.171176 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:05.239660 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:05.239626 6720 generic.go:356] "Generic (PLEG): container finished" podID="450c5ec5-0a77-449d-a515-1aa21776a572" containerID="a2280742c1687520374c40f9e9eb2b7a0374bddacb7a65d6fef0d5b788a23851" exitCode=0 Oct 30 21:58:05.240134 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:05.239684 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-689a2563/e2e-lifecycle-model-689a2563-kserve-5bb4c5b87c-q6gl8" Oct 30 21:58:05.240134 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:05.239695 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-689a2563/e2e-lifecycle-model-689a2563-kserve-5bb4c5b87c-q6gl8" event={"ID":"450c5ec5-0a77-449d-a515-1aa21776a572","Type":"ContainerDied","Data":"a2280742c1687520374c40f9e9eb2b7a0374bddacb7a65d6fef0d5b788a23851"} Oct 30 21:58:05.240134 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:05.239727 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-689a2563/e2e-lifecycle-model-689a2563-kserve-5bb4c5b87c-q6gl8" event={"ID":"450c5ec5-0a77-449d-a515-1aa21776a572","Type":"ContainerDied","Data":"4e4c9b43bd7485c9afa6233ffebf0ec06c8be06e90c581c45e241de8e02d36ef"} Oct 30 21:58:05.240134 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:05.239743 6720 scope.go:122] "RemoveContainer" containerID="a2280742c1687520374c40f9e9eb2b7a0374bddacb7a65d6fef0d5b788a23851" Oct 30 21:58:05.252776 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:05.252758 6720 scope.go:122] "RemoveContainer" containerID="a2280742c1687520374c40f9e9eb2b7a0374bddacb7a65d6fef0d5b788a23851" Oct 30 21:58:05.253058 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:58:05.253031 6720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2280742c1687520374c40f9e9eb2b7a0374bddacb7a65d6fef0d5b788a23851\": container with ID starting with a2280742c1687520374c40f9e9eb2b7a0374bddacb7a65d6fef0d5b788a23851 not found: ID does not exist" containerID="a2280742c1687520374c40f9e9eb2b7a0374bddacb7a65d6fef0d5b788a23851" Oct 30 21:58:05.253152 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:05.253086 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2280742c1687520374c40f9e9eb2b7a0374bddacb7a65d6fef0d5b788a23851"} err="failed to get container status \"a2280742c1687520374c40f9e9eb2b7a0374bddacb7a65d6fef0d5b788a23851\": rpc error: code = NotFound desc = could not find container \"a2280742c1687520374c40f9e9eb2b7a0374bddacb7a65d6fef0d5b788a23851\": container with ID starting with a2280742c1687520374c40f9e9eb2b7a0374bddacb7a65d6fef0d5b788a23851 not found: ID does not exist" Oct 30 21:58:05.274351 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:05.274327 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-689a2563/e2e-lifecycle-model-689a2563-kserve-5bb4c5b87c-q6gl8"] Oct 30 21:58:05.276815 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:05.276796 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-689a2563/e2e-lifecycle-model-689a2563-kserve-5bb4c5b87c-q6gl8"] Oct 30 21:58:05.326826 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:05.326802 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:05.880452 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:05.879419 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:05.978408 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:05.978017 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="450c5ec5-0a77-449d-a515-1aa21776a572" path="/var/lib/kubelet/pods/450c5ec5-0a77-449d-a515-1aa21776a572/volumes" Oct 30 21:58:06.272584 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:06.272498 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:06.444536 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:06.444508 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:07.156340 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:07.156300 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:07.858292 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:07.858261 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:08.703780 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:08.703744 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:09.758335 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:09.758249 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:10.321410 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:10.321356 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:10.514408 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:58:10.511086 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 21:58:10.525902 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:10.525867 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:11.230666 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:11.230499 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:11.861778 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:11.861740 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:12.042142 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:12.042109 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:12.190282 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:12.190196 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:12.371224 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:12.371189 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:13.001212 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:13.001181 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:13.225418 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:13.223520 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:14.051402 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:14.051352 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:14.831292 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:14.831258 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:14.989335 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:14.989309 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:15.393414 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:15.377039 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:15.632223 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:15.632178 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:15.848870 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:15.848770 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:16.562281 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:16.562250 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-004c17-5cbd6474d5-vk6q8"] Oct 30 21:58:16.562879 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:16.562860 6720 cpu_manager.go:414] "containerMap: removing container" podUID="450c5ec5-0a77-449d-a515-1aa21776a572" containerName="main" Oct 30 21:58:16.562879 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:16.562879 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="450c5ec5-0a77-449d-a515-1aa21776a572" containerName="main" Oct 30 21:58:16.563012 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:16.562913 6720 cpu_manager.go:414] "containerMap: removing container" podUID="7e7456b2-3efc-42fe-9c0d-9889d89ca562" containerName="maas-api" Oct 30 21:58:16.563012 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:16.562919 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e7456b2-3efc-42fe-9c0d-9889d89ca562" containerName="maas-api" Oct 30 21:58:16.563012 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:16.562931 6720 cpu_manager.go:414] "containerMap: removing container" podUID="1a924070-ecb2-4f0a-94f2-a280a470bd55" containerName="maas-api" Oct 30 21:58:16.563012 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:16.562936 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a924070-ecb2-4f0a-94f2-a280a470bd55" containerName="maas-api" Oct 30 21:58:16.563136 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:16.563026 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e7456b2-3efc-42fe-9c0d-9889d89ca562" containerName="maas-api" Oct 30 21:58:16.563136 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:16.563034 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a924070-ecb2-4f0a-94f2-a280a470bd55" containerName="maas-api" Oct 30 21:58:16.563136 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:16.563043 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="450c5ec5-0a77-449d-a515-1aa21776a572" containerName="main" Oct 30 21:58:16.568197 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:16.568167 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-004c17-5cbd6474d5-vk6q8" Oct 30 21:58:16.577374 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:16.577349 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-004c17-serving-cert\"" Oct 30 21:58:16.594997 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:16.594972 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-004c17-5cbd6474d5-vk6q8"] Oct 30 21:58:16.609217 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:16.609190 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3f95300-25a6-4d1b-8d1d-97d878caa69c-maas-api-tls\") pod \"maas-api-e2e-api-a-004c17-5cbd6474d5-vk6q8\" (UID: \"d3f95300-25a6-4d1b-8d1d-97d878caa69c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-004c17-5cbd6474d5-vk6q8" Oct 30 21:58:16.609372 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:16.609224 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3f95300-25a6-4d1b-8d1d-97d878caa69c-openshift-service-ca\") pod \"maas-api-e2e-api-a-004c17-5cbd6474d5-vk6q8\" (UID: \"d3f95300-25a6-4d1b-8d1d-97d878caa69c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-004c17-5cbd6474d5-vk6q8" Oct 30 21:58:16.609372 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:16.609321 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4sfk\" (UniqueName: \"kubernetes.io/projected/d3f95300-25a6-4d1b-8d1d-97d878caa69c-kube-api-access-c4sfk\") pod \"maas-api-e2e-api-a-004c17-5cbd6474d5-vk6q8\" (UID: \"d3f95300-25a6-4d1b-8d1d-97d878caa69c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-004c17-5cbd6474d5-vk6q8" Oct 30 21:58:16.609372 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:16.609368 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3f95300-25a6-4d1b-8d1d-97d878caa69c-metrics-certs\") pod \"maas-api-e2e-api-a-004c17-5cbd6474d5-vk6q8\" (UID: \"d3f95300-25a6-4d1b-8d1d-97d878caa69c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-004c17-5cbd6474d5-vk6q8" Oct 30 21:58:16.710920 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:16.710787 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4sfk\" (UniqueName: \"kubernetes.io/projected/d3f95300-25a6-4d1b-8d1d-97d878caa69c-kube-api-access-c4sfk\") pod \"maas-api-e2e-api-a-004c17-5cbd6474d5-vk6q8\" (UID: \"d3f95300-25a6-4d1b-8d1d-97d878caa69c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-004c17-5cbd6474d5-vk6q8" Oct 30 21:58:16.711307 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:16.711269 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3f95300-25a6-4d1b-8d1d-97d878caa69c-metrics-certs\") pod \"maas-api-e2e-api-a-004c17-5cbd6474d5-vk6q8\" (UID: \"d3f95300-25a6-4d1b-8d1d-97d878caa69c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-004c17-5cbd6474d5-vk6q8" Oct 30 21:58:16.712237 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:16.712209 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3f95300-25a6-4d1b-8d1d-97d878caa69c-maas-api-tls\") pod \"maas-api-e2e-api-a-004c17-5cbd6474d5-vk6q8\" (UID: \"d3f95300-25a6-4d1b-8d1d-97d878caa69c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-004c17-5cbd6474d5-vk6q8" Oct 30 21:58:16.712366 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:16.712246 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3f95300-25a6-4d1b-8d1d-97d878caa69c-openshift-service-ca\") pod \"maas-api-e2e-api-a-004c17-5cbd6474d5-vk6q8\" (UID: \"d3f95300-25a6-4d1b-8d1d-97d878caa69c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-004c17-5cbd6474d5-vk6q8" Oct 30 21:58:16.713305 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:16.713277 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3f95300-25a6-4d1b-8d1d-97d878caa69c-openshift-service-ca\") pod \"maas-api-e2e-api-a-004c17-5cbd6474d5-vk6q8\" (UID: \"d3f95300-25a6-4d1b-8d1d-97d878caa69c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-004c17-5cbd6474d5-vk6q8" Oct 30 21:58:16.718761 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:16.718733 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3f95300-25a6-4d1b-8d1d-97d878caa69c-maas-api-tls\") pod \"maas-api-e2e-api-a-004c17-5cbd6474d5-vk6q8\" (UID: \"d3f95300-25a6-4d1b-8d1d-97d878caa69c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-004c17-5cbd6474d5-vk6q8" Oct 30 21:58:16.724064 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:16.724041 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3f95300-25a6-4d1b-8d1d-97d878caa69c-metrics-certs\") pod \"maas-api-e2e-api-a-004c17-5cbd6474d5-vk6q8\" (UID: \"d3f95300-25a6-4d1b-8d1d-97d878caa69c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-004c17-5cbd6474d5-vk6q8" Oct 30 21:58:16.752752 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:16.752726 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4sfk\" (UniqueName: \"kubernetes.io/projected/d3f95300-25a6-4d1b-8d1d-97d878caa69c-kube-api-access-c4sfk\") pod \"maas-api-e2e-api-a-004c17-5cbd6474d5-vk6q8\" (UID: \"d3f95300-25a6-4d1b-8d1d-97d878caa69c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-004c17-5cbd6474d5-vk6q8" Oct 30 21:58:16.808125 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:16.808093 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:16.882428 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:16.882325 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-004c17-5cbd6474d5-vk6q8" Oct 30 21:58:17.154952 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:17.150207 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:17.219020 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:17.218991 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-004c17-5cbd6474d5-vk6q8"] Oct 30 21:58:17.222563 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:58:17.222534 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3f95300_25a6_4d1b_8d1d_97d878caa69c.slice/crio-92bd6d4b4695e44c9ea445abeec68750be3694d670a9b159c3878826c7d41c5a WatchSource:0}: Error finding container 92bd6d4b4695e44c9ea445abeec68750be3694d670a9b159c3878826c7d41c5a: Status 404 returned error can't find the container with id 92bd6d4b4695e44c9ea445abeec68750be3694d670a9b159c3878826c7d41c5a Oct 30 21:58:17.413633 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:17.413534 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-004c17-5cbd6474d5-vk6q8" event={"ID":"d3f95300-25a6-4d1b-8d1d-97d878caa69c","Type":"ContainerStarted","Data":"92bd6d4b4695e44c9ea445abeec68750be3694d670a9b159c3878826c7d41c5a"} Oct 30 21:58:17.744116 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:17.743929 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:17.940407 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:17.936721 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:18.145168 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:18.145135 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:18.307632 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:18.307599 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:18.421953 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:18.421867 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-004c17-5cbd6474d5-vk6q8" event={"ID":"d3f95300-25a6-4d1b-8d1d-97d878caa69c","Type":"ContainerStarted","Data":"674e1bcdae52c816bbdc5c272bb1bd81b61d079e38ee8ee16e626ed3fa7db076"} Oct 30 21:58:18.422104 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:18.422009 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-004c17-5cbd6474d5-vk6q8" Oct 30 21:58:18.450547 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:18.450495 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-004c17-5cbd6474d5-vk6q8" podStartSLOduration=1.908044344 podStartE2EDuration="2.450478632s" totalImagesPullingTime="542.434288ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:18.447921988 +0000 UTC m=+1064.618887172" watchObservedRunningTime="2026-10-30 21:58:18.450478632 +0000 UTC m=+1064.621443817" Oct 30 21:58:18.686644 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:18.686567 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:19.154966 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:19.154931 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:19.430281 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:19.430206 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:19.617601 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:19.617562 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:20.065507 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:20.065352 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:20.683123 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:20.683097 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:20.849874 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:20.849836 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:20.999949 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:20.999866 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:21.553446 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:21.553417 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:22.123259 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:22.123227 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:22.324009 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:22.323980 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:22.496225 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:22.496127 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:22.652070 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:22.652039 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:22.840527 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:22.840440 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:23.335886 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:23.335858 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:24.433678 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:24.433652 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-004c17-5cbd6474d5-vk6q8" Oct 30 21:58:24.584890 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:24.584829 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:25.482543 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:25.482508 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:26.391921 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:26.391852 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:26.558019 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:26.557987 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:26.936013 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:26.935930 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:27.118452 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:27.118398 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:27.341186 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:27.341098 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:27.581146 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:27.581107 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:27.984545 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:27.984510 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:28.203522 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:28.203490 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:28.402713 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:28.402677 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:29.401141 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:29.401109 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:30.183661 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:30.183631 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:30.363996 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:30.363966 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:30.591577 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:30.591502 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:30.912554 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:30.912524 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:31.297478 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:31.297051 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:31.479079 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:31.479046 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:31.653221 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:31.653190 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:32.107619 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:32.107584 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:32.293237 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:32.293203 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:32.691338 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:32.691295 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:33.145641 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:33.145610 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:33.713044 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:33.713003 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:34.047787 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:34.047654 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:34.485116 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:34.485087 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:34.726238 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:34.726202 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:35.381184 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:35.381147 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:35.916660 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:35.916626 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:36.128958 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:36.128923 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:36.453076 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:36.453041 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:37.799258 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:37.799222 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:38.106317 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:38.106242 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:38.486497 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:38.486376 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:39.058033 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:39.058006 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:39.229031 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:39.228995 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:39.547211 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:39.547177 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:39.895937 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:39.895900 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:40.101099 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:40.100241 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:40.416500 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:58:40.416456 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 21:58:40.568805 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:40.568750 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:40.737777 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:40.737694 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:40.907136 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:40.907109 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:41.389588 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:41.389520 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:41.608244 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:41.608213 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:41.966742 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:41.966712 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:42.142249 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:42.142217 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:42.399885 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:42.399857 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:42.750964 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:42.750881 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:42.946465 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:42.946427 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:43.144296 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:43.144261 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:43.817197 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:43.817169 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:44.845180 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:44.845146 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:44.992834 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:44.992797 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:45.207658 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:45.207630 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:45.366402 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:45.366302 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:46.045787 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:46.045754 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:46.672116 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:46.672081 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:46.813096 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:46.813060 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3654d5-fc45498c9-9tgck"] Oct 30 21:58:46.820700 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:46.820671 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3654d5-fc45498c9-9tgck" Oct 30 21:58:46.824068 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:46.824041 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:46.839102 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:46.839075 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3654d5-fc45498c9-9tgck"] Oct 30 21:58:46.932454 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:46.926359 6720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-3654d5\"" Oct 30 21:58:46.932454 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:46.926528 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-3654d5-dockercfg-n8k49\"" Oct 30 21:58:46.973465 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:46.973031 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b54e2d16-31f2-4c81-8dab-12e86f00f79f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3654d5-fc45498c9-9tgck\" (UID: \"b54e2d16-31f2-4c81-8dab-12e86f00f79f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3654d5-fc45498c9-9tgck" Oct 30 21:58:46.974033 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:46.973994 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmfz8\" (UniqueName: \"kubernetes.io/projected/b54e2d16-31f2-4c81-8dab-12e86f00f79f-kube-api-access-tmfz8\") pod \"payload-pre-processing-e2e-ipp-a-3654d5-fc45498c9-9tgck\" (UID: \"b54e2d16-31f2-4c81-8dab-12e86f00f79f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3654d5-fc45498c9-9tgck" Oct 30 21:58:47.075375 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:47.075331 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b54e2d16-31f2-4c81-8dab-12e86f00f79f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3654d5-fc45498c9-9tgck\" (UID: \"b54e2d16-31f2-4c81-8dab-12e86f00f79f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3654d5-fc45498c9-9tgck" Oct 30 21:58:47.075878 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:47.075576 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmfz8\" (UniqueName: \"kubernetes.io/projected/b54e2d16-31f2-4c81-8dab-12e86f00f79f-kube-api-access-tmfz8\") pod \"payload-pre-processing-e2e-ipp-a-3654d5-fc45498c9-9tgck\" (UID: \"b54e2d16-31f2-4c81-8dab-12e86f00f79f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3654d5-fc45498c9-9tgck" Oct 30 21:58:47.076868 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:47.076841 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b54e2d16-31f2-4c81-8dab-12e86f00f79f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3654d5-fc45498c9-9tgck\" (UID: \"b54e2d16-31f2-4c81-8dab-12e86f00f79f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3654d5-fc45498c9-9tgck" Oct 30 21:58:47.106005 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:47.105972 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmfz8\" (UniqueName: \"kubernetes.io/projected/b54e2d16-31f2-4c81-8dab-12e86f00f79f-kube-api-access-tmfz8\") pod \"payload-pre-processing-e2e-ipp-a-3654d5-fc45498c9-9tgck\" (UID: \"b54e2d16-31f2-4c81-8dab-12e86f00f79f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3654d5-fc45498c9-9tgck" Oct 30 21:58:47.108684 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:47.108615 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:47.144067 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:47.144029 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3654d5-fc45498c9-9tgck" Oct 30 21:58:47.286454 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:47.283406 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:47.478066 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:47.478036 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3654d5-fc45498c9-9tgck"] Oct 30 21:58:47.480476 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:58:47.480440 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb54e2d16_31f2_4c81_8dab_12e86f00f79f.slice/crio-4f0c183474a9062da6b1bb8bfe9f649b216d58a5236bbc41a75f347a161bf262 WatchSource:0}: Error finding container 4f0c183474a9062da6b1bb8bfe9f649b216d58a5236bbc41a75f347a161bf262: Status 404 returned error can't find the container with id 4f0c183474a9062da6b1bb8bfe9f649b216d58a5236bbc41a75f347a161bf262 Oct 30 21:58:47.631245 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:47.629494 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:47.670535 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:47.670503 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3654d5-fc45498c9-9tgck" event={"ID":"b54e2d16-31f2-4c81-8dab-12e86f00f79f","Type":"ContainerStarted","Data":"4f0c183474a9062da6b1bb8bfe9f649b216d58a5236bbc41a75f347a161bf262"} Oct 30 21:58:48.567576 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:48.567511 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:48.754375 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:48.754342 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:49.695921 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:49.695886 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3654d5-fc45498c9-9tgck" event={"ID":"b54e2d16-31f2-4c81-8dab-12e86f00f79f","Type":"ContainerStarted","Data":"d549fce84e231092ace93948567579bab651132b43598c13ee5b62ed48908f90"} Oct 30 21:58:49.696337 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:49.695950 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3654d5-fc45498c9-9tgck" Oct 30 21:58:49.720206 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:49.720173 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:49.721530 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:49.721487 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3654d5-fc45498c9-9tgck" podStartSLOduration=2.07791912 podStartE2EDuration="3.721474053s" totalImagesPullingTime="1.643554933s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:49.71592003 +0000 UTC m=+1095.886885214" watchObservedRunningTime="2026-10-30 21:58:49.721474053 +0000 UTC m=+1095.892439235" Oct 30 21:58:50.370609 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:50.370574 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:51.226972 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:51.226943 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:51.811042 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:51.811006 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:52.029411 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:52.008854 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:52.182496 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:52.180459 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:52.537521 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:52.537351 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:53.150135 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:53.150103 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:54.102901 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:54.102872 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:54.800755 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:54.800722 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e3670-6fb95fbcb8-zbsns"] Oct 30 21:58:54.805160 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:54.805138 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e3670-6fb95fbcb8-zbsns" Oct 30 21:58:54.829137 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:54.829115 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-3e3670-serving-cert\"" Oct 30 21:58:54.848786 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:54.848763 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e3670-6fb95fbcb8-zbsns"] Oct 30 21:58:54.904399 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:54.904353 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:54.913234 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:54.913204 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/907c3a08-898e-433a-8ec6-3ee3fceb949e-maas-api-tls\") pod \"maas-api-e2e-ipp-b-3e3670-6fb95fbcb8-zbsns\" (UID: \"907c3a08-898e-433a-8ec6-3ee3fceb949e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e3670-6fb95fbcb8-zbsns" Oct 30 21:58:54.913378 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:54.913244 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/907c3a08-898e-433a-8ec6-3ee3fceb949e-metrics-certs\") pod \"maas-api-e2e-ipp-b-3e3670-6fb95fbcb8-zbsns\" (UID: \"907c3a08-898e-433a-8ec6-3ee3fceb949e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e3670-6fb95fbcb8-zbsns" Oct 30 21:58:54.913378 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:54.913327 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/907c3a08-898e-433a-8ec6-3ee3fceb949e-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-3e3670-6fb95fbcb8-zbsns\" (UID: \"907c3a08-898e-433a-8ec6-3ee3fceb949e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e3670-6fb95fbcb8-zbsns" Oct 30 21:58:54.913527 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:54.913420 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pj9hq\" (UniqueName: \"kubernetes.io/projected/907c3a08-898e-433a-8ec6-3ee3fceb949e-kube-api-access-pj9hq\") pod \"maas-api-e2e-ipp-b-3e3670-6fb95fbcb8-zbsns\" (UID: \"907c3a08-898e-433a-8ec6-3ee3fceb949e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e3670-6fb95fbcb8-zbsns" Oct 30 21:58:55.014965 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:55.014930 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/907c3a08-898e-433a-8ec6-3ee3fceb949e-maas-api-tls\") pod \"maas-api-e2e-ipp-b-3e3670-6fb95fbcb8-zbsns\" (UID: \"907c3a08-898e-433a-8ec6-3ee3fceb949e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e3670-6fb95fbcb8-zbsns" Oct 30 21:58:55.014965 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:55.014969 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/907c3a08-898e-433a-8ec6-3ee3fceb949e-metrics-certs\") pod \"maas-api-e2e-ipp-b-3e3670-6fb95fbcb8-zbsns\" (UID: \"907c3a08-898e-433a-8ec6-3ee3fceb949e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e3670-6fb95fbcb8-zbsns" Oct 30 21:58:55.015208 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:55.014993 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/907c3a08-898e-433a-8ec6-3ee3fceb949e-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-3e3670-6fb95fbcb8-zbsns\" (UID: \"907c3a08-898e-433a-8ec6-3ee3fceb949e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e3670-6fb95fbcb8-zbsns" Oct 30 21:58:55.015208 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:55.015039 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj9hq\" (UniqueName: \"kubernetes.io/projected/907c3a08-898e-433a-8ec6-3ee3fceb949e-kube-api-access-pj9hq\") pod \"maas-api-e2e-ipp-b-3e3670-6fb95fbcb8-zbsns\" (UID: \"907c3a08-898e-433a-8ec6-3ee3fceb949e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e3670-6fb95fbcb8-zbsns" Oct 30 21:58:55.015840 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:55.015817 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/907c3a08-898e-433a-8ec6-3ee3fceb949e-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-3e3670-6fb95fbcb8-zbsns\" (UID: \"907c3a08-898e-433a-8ec6-3ee3fceb949e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e3670-6fb95fbcb8-zbsns" Oct 30 21:58:55.017785 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:55.017756 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/907c3a08-898e-433a-8ec6-3ee3fceb949e-metrics-certs\") pod \"maas-api-e2e-ipp-b-3e3670-6fb95fbcb8-zbsns\" (UID: \"907c3a08-898e-433a-8ec6-3ee3fceb949e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e3670-6fb95fbcb8-zbsns" Oct 30 21:58:55.017885 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:55.017862 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/907c3a08-898e-433a-8ec6-3ee3fceb949e-maas-api-tls\") pod \"maas-api-e2e-ipp-b-3e3670-6fb95fbcb8-zbsns\" (UID: \"907c3a08-898e-433a-8ec6-3ee3fceb949e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e3670-6fb95fbcb8-zbsns" Oct 30 21:58:55.030118 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:55.030095 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj9hq\" (UniqueName: \"kubernetes.io/projected/907c3a08-898e-433a-8ec6-3ee3fceb949e-kube-api-access-pj9hq\") pod \"maas-api-e2e-ipp-b-3e3670-6fb95fbcb8-zbsns\" (UID: \"907c3a08-898e-433a-8ec6-3ee3fceb949e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e3670-6fb95fbcb8-zbsns" Oct 30 21:58:55.119230 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:55.119202 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e3670-6fb95fbcb8-zbsns" Oct 30 21:58:55.141961 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:55.141932 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:55.423813 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:55.423782 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e3670-6fb95fbcb8-zbsns"] Oct 30 21:58:55.740935 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:55.740851 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e3670-6fb95fbcb8-zbsns" event={"ID":"907c3a08-898e-433a-8ec6-3ee3fceb949e","Type":"ContainerStarted","Data":"abfa203ad20248dffa180ae11601fa0c13f7225de8165db0106597ad946bd0c5"} Oct 30 21:58:55.937317 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:55.937291 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:56.749816 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:56.749768 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e3670-6fb95fbcb8-zbsns" event={"ID":"907c3a08-898e-433a-8ec6-3ee3fceb949e","Type":"ContainerStarted","Data":"3140dad1fc7dbfe5027ff84bd52ebe10c573bc7af383a1cbdbfc27cfab042f85"} Oct 30 21:58:56.750327 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:56.749878 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e3670-6fb95fbcb8-zbsns" Oct 30 21:58:56.776175 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:56.776116 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e3670-6fb95fbcb8-zbsns" podStartSLOduration=2.347230059 podStartE2EDuration="2.776099525s" totalImagesPullingTime="428.869466ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:56.775250064 +0000 UTC m=+1102.946215247" watchObservedRunningTime="2026-10-30 21:58:56.776099525 +0000 UTC m=+1102.947064709" Oct 30 21:58:56.944135 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:56.944098 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:57.905964 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:57.905934 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:58.128046 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:58.128014 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:58.289096 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:58.288998 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:58.779346 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:58.779305 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:59.568373 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:59.568340 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:59.733349 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:59.733319 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:58:59.961177 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:58:59.961138 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:00.706916 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:00.706884 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3654d5-fc45498c9-9tgck" Oct 30 21:59:00.747802 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:00.747769 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:00.961352 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:00.961279 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:01.240491 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:01.240401 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:01.451926 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:01.451880 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:01.608511 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:01.608461 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:01.897442 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:01.897315 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:02.762498 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:02.762472 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e3670-6fb95fbcb8-zbsns" Oct 30 21:59:02.932028 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:02.931999 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:03.208019 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:03.207988 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:03.366799 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:03.366763 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:03.531528 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:03.531427 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:03.872240 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:03.872169 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:04.921561 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:04.921508 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:05.168114 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:05.168081 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:05.379042 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:05.379010 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:06.038767 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:06.038720 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:07.269940 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:07.269904 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:07.881235 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:07.881187 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:08.835753 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:08.835712 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:09.749028 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:09.748933 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:10.415719 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:59:10.415677 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 21:59:10.494734 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:10.494698 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:10.823336 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:10.823249 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:11.239631 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:11.239598 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:11.398140 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:11.398107 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:11.839746 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:11.839709 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:12.078535 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:12.078489 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:12.228364 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:12.228331 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:12.449028 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:12.448999 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:12.612424 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:12.612355 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:12.894104 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:12.894005 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:13.135053 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:13.135023 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:13.628370 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:13.628338 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:14.506675 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:14.506641 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:14.771584 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:14.771490 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:15.641999 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:15.641970 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:16.244859 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:16.244826 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c614f9-69b7c4fb64-kgzkp"] Oct 30 21:59:16.251015 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:16.250993 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c614f9-69b7c4fb64-kgzkp" Oct 30 21:59:16.257900 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:16.257875 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-c614f9-serving-cert\"" Oct 30 21:59:16.261826 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:16.261783 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c614f9-69b7c4fb64-kgzkp"] Oct 30 21:59:16.336136 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:16.336107 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggqm2\" (UniqueName: \"kubernetes.io/projected/5f8bf96d-b5df-4775-9d77-a725a0ca6a2b-kube-api-access-ggqm2\") pod \"maas-api-e2e-ipp-a-c614f9-69b7c4fb64-kgzkp\" (UID: \"5f8bf96d-b5df-4775-9d77-a725a0ca6a2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c614f9-69b7c4fb64-kgzkp" Oct 30 21:59:16.336294 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:16.336146 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f8bf96d-b5df-4775-9d77-a725a0ca6a2b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c614f9-69b7c4fb64-kgzkp\" (UID: \"5f8bf96d-b5df-4775-9d77-a725a0ca6a2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c614f9-69b7c4fb64-kgzkp" Oct 30 21:59:16.336433 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:16.336369 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f8bf96d-b5df-4775-9d77-a725a0ca6a2b-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-c614f9-69b7c4fb64-kgzkp\" (UID: \"5f8bf96d-b5df-4775-9d77-a725a0ca6a2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c614f9-69b7c4fb64-kgzkp" Oct 30 21:59:16.336476 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:16.336466 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f8bf96d-b5df-4775-9d77-a725a0ca6a2b-metrics-certs\") pod \"maas-api-e2e-ipp-a-c614f9-69b7c4fb64-kgzkp\" (UID: \"5f8bf96d-b5df-4775-9d77-a725a0ca6a2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c614f9-69b7c4fb64-kgzkp" Oct 30 21:59:16.397001 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:16.396969 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:16.437883 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:16.437850 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggqm2\" (UniqueName: \"kubernetes.io/projected/5f8bf96d-b5df-4775-9d77-a725a0ca6a2b-kube-api-access-ggqm2\") pod \"maas-api-e2e-ipp-a-c614f9-69b7c4fb64-kgzkp\" (UID: \"5f8bf96d-b5df-4775-9d77-a725a0ca6a2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c614f9-69b7c4fb64-kgzkp" Oct 30 21:59:16.438070 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:16.437894 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f8bf96d-b5df-4775-9d77-a725a0ca6a2b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c614f9-69b7c4fb64-kgzkp\" (UID: \"5f8bf96d-b5df-4775-9d77-a725a0ca6a2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c614f9-69b7c4fb64-kgzkp" Oct 30 21:59:16.438070 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:16.438054 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f8bf96d-b5df-4775-9d77-a725a0ca6a2b-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-c614f9-69b7c4fb64-kgzkp\" (UID: \"5f8bf96d-b5df-4775-9d77-a725a0ca6a2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c614f9-69b7c4fb64-kgzkp" Oct 30 21:59:16.438174 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:59:16.438078 6720 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-c614f9-serving-cert: secret "maas-api-e2e-ipp-a-c614f9-serving-cert" not found Oct 30 21:59:16.438174 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:59:16.438141 6720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f8bf96d-b5df-4775-9d77-a725a0ca6a2b-maas-api-tls podName:5f8bf96d-b5df-4775-9d77-a725a0ca6a2b nodeName:}" failed. No retries permitted until 2026-10-30 21:59:16.938119173 +0000 UTC m=+1123.109084350 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5f8bf96d-b5df-4775-9d77-a725a0ca6a2b-maas-api-tls") pod "maas-api-e2e-ipp-a-c614f9-69b7c4fb64-kgzkp" (UID: "5f8bf96d-b5df-4775-9d77-a725a0ca6a2b") : secret "maas-api-e2e-ipp-a-c614f9-serving-cert" not found Oct 30 21:59:16.438174 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:16.438157 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f8bf96d-b5df-4775-9d77-a725a0ca6a2b-metrics-certs\") pod \"maas-api-e2e-ipp-a-c614f9-69b7c4fb64-kgzkp\" (UID: \"5f8bf96d-b5df-4775-9d77-a725a0ca6a2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c614f9-69b7c4fb64-kgzkp" Oct 30 21:59:16.438821 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:16.438796 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f8bf96d-b5df-4775-9d77-a725a0ca6a2b-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-c614f9-69b7c4fb64-kgzkp\" (UID: \"5f8bf96d-b5df-4775-9d77-a725a0ca6a2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c614f9-69b7c4fb64-kgzkp" Oct 30 21:59:16.441117 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:16.441089 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f8bf96d-b5df-4775-9d77-a725a0ca6a2b-metrics-certs\") pod \"maas-api-e2e-ipp-a-c614f9-69b7c4fb64-kgzkp\" (UID: \"5f8bf96d-b5df-4775-9d77-a725a0ca6a2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c614f9-69b7c4fb64-kgzkp" Oct 30 21:59:16.451376 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:16.451352 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggqm2\" (UniqueName: \"kubernetes.io/projected/5f8bf96d-b5df-4775-9d77-a725a0ca6a2b-kube-api-access-ggqm2\") pod \"maas-api-e2e-ipp-a-c614f9-69b7c4fb64-kgzkp\" (UID: \"5f8bf96d-b5df-4775-9d77-a725a0ca6a2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c614f9-69b7c4fb64-kgzkp" Oct 30 21:59:16.602331 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:16.602243 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:16.814866 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:16.814835 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:16.944492 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:16.944464 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f8bf96d-b5df-4775-9d77-a725a0ca6a2b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c614f9-69b7c4fb64-kgzkp\" (UID: \"5f8bf96d-b5df-4775-9d77-a725a0ca6a2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c614f9-69b7c4fb64-kgzkp" Oct 30 21:59:16.947627 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:16.947593 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f8bf96d-b5df-4775-9d77-a725a0ca6a2b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c614f9-69b7c4fb64-kgzkp\" (UID: \"5f8bf96d-b5df-4775-9d77-a725a0ca6a2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c614f9-69b7c4fb64-kgzkp" Oct 30 21:59:17.171655 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:17.171622 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c614f9-69b7c4fb64-kgzkp" Oct 30 21:59:17.470982 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:17.470951 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:17.471599 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:17.471577 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c614f9-69b7c4fb64-kgzkp"] Oct 30 21:59:17.785662 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:17.785632 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:17.937975 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:17.937942 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c614f9-69b7c4fb64-kgzkp" event={"ID":"5f8bf96d-b5df-4775-9d77-a725a0ca6a2b","Type":"ContainerStarted","Data":"bc71e985ac5dbe392b547a79cf836d13c9f5e9b2fffefbc0c4a4fbb0b26b2b09"} Oct 30 21:59:17.938805 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:17.938784 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:18.615840 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:18.615795 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:18.863976 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:18.863942 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rmpbr"] Oct 30 21:59:18.868698 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:18.868625 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rmpbr" Oct 30 21:59:18.887254 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:18.887223 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rmpbr"] Oct 30 21:59:18.950900 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:18.950860 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c614f9-69b7c4fb64-kgzkp" event={"ID":"5f8bf96d-b5df-4775-9d77-a725a0ca6a2b","Type":"ContainerStarted","Data":"83c1cdc4f3de904a53626c55b6e1281e005d12be3f7001cf5c44da27d451bf09"} Oct 30 21:59:18.951359 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:18.951016 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c614f9-69b7c4fb64-kgzkp" Oct 30 21:59:18.975537 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:18.975491 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c614f9-69b7c4fb64-kgzkp" podStartSLOduration=2.464357523 podStartE2EDuration="2.975476651s" totalImagesPullingTime="511.119128ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:18.972067482 +0000 UTC m=+1125.143032666" watchObservedRunningTime="2026-10-30 21:59:18.975476651 +0000 UTC m=+1125.146441863" Oct 30 21:59:18.995040 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:18.995013 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47ca938c-cc9e-45b3-95c1-1f9cf1206316-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rmpbr\" (UID: \"47ca938c-cc9e-45b3-95c1-1f9cf1206316\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rmpbr" Oct 30 21:59:18.995183 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:18.995076 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/47ca938c-cc9e-45b3-95c1-1f9cf1206316-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rmpbr\" (UID: \"47ca938c-cc9e-45b3-95c1-1f9cf1206316\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rmpbr" Oct 30 21:59:18.995183 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:18.995169 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/47ca938c-cc9e-45b3-95c1-1f9cf1206316-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rmpbr\" (UID: \"47ca938c-cc9e-45b3-95c1-1f9cf1206316\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rmpbr" Oct 30 21:59:18.995277 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:18.995267 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz2r9\" (UniqueName: \"kubernetes.io/projected/47ca938c-cc9e-45b3-95c1-1f9cf1206316-kube-api-access-fz2r9\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rmpbr\" (UID: \"47ca938c-cc9e-45b3-95c1-1f9cf1206316\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rmpbr" Oct 30 21:59:18.995314 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:18.995299 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/47ca938c-cc9e-45b3-95c1-1f9cf1206316-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rmpbr\" (UID: \"47ca938c-cc9e-45b3-95c1-1f9cf1206316\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rmpbr" Oct 30 21:59:18.995370 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:18.995322 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/47ca938c-cc9e-45b3-95c1-1f9cf1206316-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rmpbr\" (UID: \"47ca938c-cc9e-45b3-95c1-1f9cf1206316\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rmpbr" Oct 30 21:59:18.995516 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:18.995497 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/47ca938c-cc9e-45b3-95c1-1f9cf1206316-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rmpbr\" (UID: \"47ca938c-cc9e-45b3-95c1-1f9cf1206316\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rmpbr" Oct 30 21:59:19.011071 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:19.011048 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:19.098131 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:19.098096 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/47ca938c-cc9e-45b3-95c1-1f9cf1206316-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rmpbr\" (UID: \"47ca938c-cc9e-45b3-95c1-1f9cf1206316\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rmpbr" Oct 30 21:59:19.098442 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:19.098224 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz2r9\" (UniqueName: \"kubernetes.io/projected/47ca938c-cc9e-45b3-95c1-1f9cf1206316-kube-api-access-fz2r9\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rmpbr\" (UID: \"47ca938c-cc9e-45b3-95c1-1f9cf1206316\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rmpbr" Oct 30 21:59:19.098442 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:19.098276 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/47ca938c-cc9e-45b3-95c1-1f9cf1206316-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rmpbr\" (UID: \"47ca938c-cc9e-45b3-95c1-1f9cf1206316\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rmpbr" Oct 30 21:59:19.098442 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:19.098308 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/47ca938c-cc9e-45b3-95c1-1f9cf1206316-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rmpbr\" (UID: \"47ca938c-cc9e-45b3-95c1-1f9cf1206316\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rmpbr" Oct 30 21:59:19.098646 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:19.098537 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/47ca938c-cc9e-45b3-95c1-1f9cf1206316-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rmpbr\" (UID: \"47ca938c-cc9e-45b3-95c1-1f9cf1206316\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rmpbr" Oct 30 21:59:19.098768 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:19.098743 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47ca938c-cc9e-45b3-95c1-1f9cf1206316-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rmpbr\" (UID: \"47ca938c-cc9e-45b3-95c1-1f9cf1206316\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rmpbr" Oct 30 21:59:19.098867 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:19.098823 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/47ca938c-cc9e-45b3-95c1-1f9cf1206316-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rmpbr\" (UID: \"47ca938c-cc9e-45b3-95c1-1f9cf1206316\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rmpbr" Oct 30 21:59:19.099523 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:19.099493 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/47ca938c-cc9e-45b3-95c1-1f9cf1206316-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rmpbr\" (UID: \"47ca938c-cc9e-45b3-95c1-1f9cf1206316\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rmpbr" Oct 30 21:59:19.101421 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:19.101360 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/47ca938c-cc9e-45b3-95c1-1f9cf1206316-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rmpbr\" (UID: \"47ca938c-cc9e-45b3-95c1-1f9cf1206316\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rmpbr" Oct 30 21:59:19.101697 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:19.101678 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/47ca938c-cc9e-45b3-95c1-1f9cf1206316-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rmpbr\" (UID: \"47ca938c-cc9e-45b3-95c1-1f9cf1206316\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rmpbr" Oct 30 21:59:19.101970 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:19.101941 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/47ca938c-cc9e-45b3-95c1-1f9cf1206316-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rmpbr\" (UID: \"47ca938c-cc9e-45b3-95c1-1f9cf1206316\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rmpbr" Oct 30 21:59:19.102066 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:19.102050 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/47ca938c-cc9e-45b3-95c1-1f9cf1206316-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rmpbr\" (UID: \"47ca938c-cc9e-45b3-95c1-1f9cf1206316\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rmpbr" Oct 30 21:59:19.119107 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:19.119044 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz2r9\" (UniqueName: \"kubernetes.io/projected/47ca938c-cc9e-45b3-95c1-1f9cf1206316-kube-api-access-fz2r9\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rmpbr\" (UID: \"47ca938c-cc9e-45b3-95c1-1f9cf1206316\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rmpbr" Oct 30 21:59:19.119548 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:19.119527 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47ca938c-cc9e-45b3-95c1-1f9cf1206316-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rmpbr\" (UID: \"47ca938c-cc9e-45b3-95c1-1f9cf1206316\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rmpbr" Oct 30 21:59:19.190886 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:19.190844 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rmpbr" Oct 30 21:59:19.194449 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:19.194421 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:19.370702 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:19.370602 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:19.562120 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:19.562091 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rmpbr"] Oct 30 21:59:19.564142 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:19.564110 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:59:19.564306 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:19.564290 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 21:59:19.829121 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:19.829088 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:19.976479 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:19.976338 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rmpbr" event={"ID":"47ca938c-cc9e-45b3-95c1-1f9cf1206316","Type":"ContainerStarted","Data":"205e5f21ab3acba6f14a6c566df0cd5435e43179be3a4a609abaef3f9c0070b8"} Oct 30 21:59:19.976479 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:19.976439 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rmpbr" event={"ID":"47ca938c-cc9e-45b3-95c1-1f9cf1206316","Type":"ContainerStarted","Data":"c9c92c57da5ff0ffa752f063562e134f1e95b1b91c87db2b5d058156ea7ece66"} Oct 30 21:59:19.976983 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:19.976483 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rmpbr" Oct 30 21:59:20.009319 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:20.009258 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rmpbr" podStartSLOduration=2.009237979 podStartE2EDuration="2.009237979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:20.000812386 +0000 UTC m=+1126.171777587" watchObservedRunningTime="2026-10-30 21:59:20.009237979 +0000 UTC m=+1126.180203172" Oct 30 21:59:20.604421 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:20.604115 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:20.984110 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:20.984044 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rmpbr" Oct 30 21:59:21.610970 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:21.610937 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:22.338703 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:22.338665 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:23.019845 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:23.019816 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:23.841987 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:23.841949 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:24.529621 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:24.529572 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:24.982660 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:24.982630 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c614f9-69b7c4fb64-kgzkp" Oct 30 21:59:25.211431 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:25.211371 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:26.037265 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:26.037230 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:26.243443 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:26.243369 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:26.761054 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:26.761019 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:27.000191 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:27.000157 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:27.555591 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:27.555554 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:27.722632 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:27.722594 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:28.001304 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:28.001261 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:28.344527 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:28.344432 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-004c17-5cbd6474d5-vk6q8"] Oct 30 21:59:28.344957 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:28.344927 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-004c17-5cbd6474d5-vk6q8" podUID="d3f95300-25a6-4d1b-8d1d-97d878caa69c" containerName="maas-api" containerID="cri-o://674e1bcdae52c816bbdc5c272bb1bd81b61d079e38ee8ee16e626ed3fa7db076" gracePeriod=30 Oct 30 21:59:28.399933 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:28.399903 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:28.565132 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:28.565101 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:28.747232 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:28.747208 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-004c17-5cbd6474d5-vk6q8" Oct 30 21:59:28.754431 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:28.754025 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:28.803322 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:28.801896 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3f95300-25a6-4d1b-8d1d-97d878caa69c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3f95300-25a6-4d1b-8d1d-97d878caa69c-metrics-certs\") pod \"d3f95300-25a6-4d1b-8d1d-97d878caa69c\" (UID: \"d3f95300-25a6-4d1b-8d1d-97d878caa69c\") " Oct 30 21:59:28.803322 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:28.801964 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3f95300-25a6-4d1b-8d1d-97d878caa69c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3f95300-25a6-4d1b-8d1d-97d878caa69c-maas-api-tls\") pod \"d3f95300-25a6-4d1b-8d1d-97d878caa69c\" (UID: \"d3f95300-25a6-4d1b-8d1d-97d878caa69c\") " Oct 30 21:59:28.803322 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:28.802151 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3f95300-25a6-4d1b-8d1d-97d878caa69c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3f95300-25a6-4d1b-8d1d-97d878caa69c-openshift-service-ca\") pod \"d3f95300-25a6-4d1b-8d1d-97d878caa69c\" (UID: \"d3f95300-25a6-4d1b-8d1d-97d878caa69c\") " Oct 30 21:59:28.803322 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:28.802194 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3f95300-25a6-4d1b-8d1d-97d878caa69c-kube-api-access-c4sfk\" (UniqueName: \"kubernetes.io/projected/d3f95300-25a6-4d1b-8d1d-97d878caa69c-kube-api-access-c4sfk\") pod \"d3f95300-25a6-4d1b-8d1d-97d878caa69c\" (UID: \"d3f95300-25a6-4d1b-8d1d-97d878caa69c\") " Oct 30 21:59:28.803691 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:28.803566 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3f95300-25a6-4d1b-8d1d-97d878caa69c-openshift-service-ca" pod "d3f95300-25a6-4d1b-8d1d-97d878caa69c" (UID: "d3f95300-25a6-4d1b-8d1d-97d878caa69c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:59:28.807210 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:28.807026 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3f95300-25a6-4d1b-8d1d-97d878caa69c-metrics-certs" pod "d3f95300-25a6-4d1b-8d1d-97d878caa69c" (UID: "d3f95300-25a6-4d1b-8d1d-97d878caa69c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:59:28.809797 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:28.809772 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3f95300-25a6-4d1b-8d1d-97d878caa69c-maas-api-tls" pod "d3f95300-25a6-4d1b-8d1d-97d878caa69c" (UID: "d3f95300-25a6-4d1b-8d1d-97d878caa69c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:59:28.811933 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:28.811902 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3f95300-25a6-4d1b-8d1d-97d878caa69c-kube-api-access-c4sfk" pod "d3f95300-25a6-4d1b-8d1d-97d878caa69c" (UID: "d3f95300-25a6-4d1b-8d1d-97d878caa69c"). InnerVolumeSpecName "kube-api-access-c4sfk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:59:28.904688 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:28.904663 6720 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3f95300-25a6-4d1b-8d1d-97d878caa69c-metrics-certs\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:28.904688 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:28.904686 6720 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3f95300-25a6-4d1b-8d1d-97d878caa69c-maas-api-tls\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:28.904688 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:28.904699 6720 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3f95300-25a6-4d1b-8d1d-97d878caa69c-openshift-service-ca\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:28.904921 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:28.904709 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4sfk\" (UniqueName: \"kubernetes.io/projected/d3f95300-25a6-4d1b-8d1d-97d878caa69c-kube-api-access-c4sfk\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 21:59:29.029137 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:29.028017 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:29.034705 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:29.034672 6720 generic.go:356] "Generic (PLEG): container finished" podID="d3f95300-25a6-4d1b-8d1d-97d878caa69c" containerID="674e1bcdae52c816bbdc5c272bb1bd81b61d079e38ee8ee16e626ed3fa7db076" exitCode=0 Oct 30 21:59:29.034880 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:29.034722 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-004c17-5cbd6474d5-vk6q8" Oct 30 21:59:29.034880 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:29.034762 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-004c17-5cbd6474d5-vk6q8" event={"ID":"d3f95300-25a6-4d1b-8d1d-97d878caa69c","Type":"ContainerDied","Data":"674e1bcdae52c816bbdc5c272bb1bd81b61d079e38ee8ee16e626ed3fa7db076"} Oct 30 21:59:29.034880 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:29.034827 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-004c17-5cbd6474d5-vk6q8" event={"ID":"d3f95300-25a6-4d1b-8d1d-97d878caa69c","Type":"ContainerDied","Data":"92bd6d4b4695e44c9ea445abeec68750be3694d670a9b159c3878826c7d41c5a"} Oct 30 21:59:29.034880 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:29.034858 6720 scope.go:122] "RemoveContainer" containerID="674e1bcdae52c816bbdc5c272bb1bd81b61d079e38ee8ee16e626ed3fa7db076" Oct 30 21:59:29.046202 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:29.046184 6720 scope.go:122] "RemoveContainer" containerID="674e1bcdae52c816bbdc5c272bb1bd81b61d079e38ee8ee16e626ed3fa7db076" Oct 30 21:59:29.046537 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:59:29.046502 6720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"674e1bcdae52c816bbdc5c272bb1bd81b61d079e38ee8ee16e626ed3fa7db076\": container with ID starting with 674e1bcdae52c816bbdc5c272bb1bd81b61d079e38ee8ee16e626ed3fa7db076 not found: ID does not exist" containerID="674e1bcdae52c816bbdc5c272bb1bd81b61d079e38ee8ee16e626ed3fa7db076" Oct 30 21:59:29.046685 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:29.046549 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"674e1bcdae52c816bbdc5c272bb1bd81b61d079e38ee8ee16e626ed3fa7db076"} err="failed to get container status \"674e1bcdae52c816bbdc5c272bb1bd81b61d079e38ee8ee16e626ed3fa7db076\": rpc error: code = NotFound desc = could not find container \"674e1bcdae52c816bbdc5c272bb1bd81b61d079e38ee8ee16e626ed3fa7db076\": container with ID starting with 674e1bcdae52c816bbdc5c272bb1bd81b61d079e38ee8ee16e626ed3fa7db076 not found: ID does not exist" Oct 30 21:59:29.098527 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:29.098495 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-004c17-5cbd6474d5-vk6q8"] Oct 30 21:59:29.106836 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:29.106797 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-004c17-5cbd6474d5-vk6q8"] Oct 30 21:59:29.243208 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:29.243170 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:29.751423 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:29.751374 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:29.958163 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:29.958132 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3f95300-25a6-4d1b-8d1d-97d878caa69c" path="/var/lib/kubelet/pods/d3f95300-25a6-4d1b-8d1d-97d878caa69c/volumes" Oct 30 21:59:30.204741 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:30.204711 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:30.764285 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:30.764242 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:30.962310 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:30.962276 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:31.140718 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:31.138902 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:31.735239 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:31.735208 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:31.903019 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:31.902986 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:32.803843 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:32.803807 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:33.041282 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:33.041250 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45110d7e-jj2v4"] Oct 30 21:59:33.046030 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:33.045998 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45110d7e-jj2v4"] Oct 30 21:59:33.923114 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:33.923070 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:33.958264 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:33.958229 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b21b427-82ce-45a6-a795-57e67ac46259" path="/var/lib/kubelet/pods/9b21b427-82ce-45a6-a795-57e67ac46259/volumes" Oct 30 21:59:35.015969 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:35.015933 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:35.619663 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:35.619632 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:35.802575 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:35.802539 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:36.143969 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:36.143943 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:36.789859 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:36.789809 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:36.969722 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:36.969677 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:37.146944 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:37.146912 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:37.633889 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:37.633853 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:38.624311 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:38.624226 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:39.383637 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:39.383591 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:39.801422 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:39.801277 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:39.951672 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:39.951636 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:40.359305 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:40.359270 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:40.403694 ip-10-0-137-181 kubenswrapper[6720]: E1030 21:59:40.403654 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 21:59:41.130848 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:41.130788 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:42.034138 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:42.034102 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:43.002882 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:43.002836 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:43.156609 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:43.156571 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:43.566108 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:43.566072 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:43.859192 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:43.859156 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:44.729696 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:44.729661 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:45.403341 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:45.403297 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:46.009887 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:46.009854 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-19cab1-5855fdfb65-zxd9n"] Oct 30 21:59:46.010573 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:46.010546 6720 cpu_manager.go:414] "containerMap: removing container" podUID="d3f95300-25a6-4d1b-8d1d-97d878caa69c" containerName="maas-api" Oct 30 21:59:46.010573 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:46.010567 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3f95300-25a6-4d1b-8d1d-97d878caa69c" containerName="maas-api" Oct 30 21:59:46.010850 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:46.010739 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3f95300-25a6-4d1b-8d1d-97d878caa69c" containerName="maas-api" Oct 30 21:59:46.016229 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:46.016205 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-19cab1-5855fdfb65-zxd9n" Oct 30 21:59:46.025866 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:46.025838 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-19cab1-serving-cert\"" Oct 30 21:59:46.031212 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:46.031188 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-19cab1-5855fdfb65-zxd9n"] Oct 30 21:59:46.037246 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:46.037215 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64sqf\" (UniqueName: \"kubernetes.io/projected/64d0f5bd-9b36-4ec4-b3c3-d47976e760e9-kube-api-access-64sqf\") pod \"maas-api-e2e-ipp-b-19cab1-5855fdfb65-zxd9n\" (UID: \"64d0f5bd-9b36-4ec4-b3c3-d47976e760e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-19cab1-5855fdfb65-zxd9n" Oct 30 21:59:46.037419 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:46.037288 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64d0f5bd-9b36-4ec4-b3c3-d47976e760e9-metrics-certs\") pod \"maas-api-e2e-ipp-b-19cab1-5855fdfb65-zxd9n\" (UID: \"64d0f5bd-9b36-4ec4-b3c3-d47976e760e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-19cab1-5855fdfb65-zxd9n" Oct 30 21:59:46.037499 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:46.037436 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64d0f5bd-9b36-4ec4-b3c3-d47976e760e9-maas-api-tls\") pod \"maas-api-e2e-ipp-b-19cab1-5855fdfb65-zxd9n\" (UID: \"64d0f5bd-9b36-4ec4-b3c3-d47976e760e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-19cab1-5855fdfb65-zxd9n" Oct 30 21:59:46.037566 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:46.037542 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64d0f5bd-9b36-4ec4-b3c3-d47976e760e9-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-19cab1-5855fdfb65-zxd9n\" (UID: \"64d0f5bd-9b36-4ec4-b3c3-d47976e760e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-19cab1-5855fdfb65-zxd9n" Oct 30 21:59:46.139092 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:46.139049 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64d0f5bd-9b36-4ec4-b3c3-d47976e760e9-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-19cab1-5855fdfb65-zxd9n\" (UID: \"64d0f5bd-9b36-4ec4-b3c3-d47976e760e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-19cab1-5855fdfb65-zxd9n" Oct 30 21:59:46.139259 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:46.139244 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64sqf\" (UniqueName: \"kubernetes.io/projected/64d0f5bd-9b36-4ec4-b3c3-d47976e760e9-kube-api-access-64sqf\") pod \"maas-api-e2e-ipp-b-19cab1-5855fdfb65-zxd9n\" (UID: \"64d0f5bd-9b36-4ec4-b3c3-d47976e760e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-19cab1-5855fdfb65-zxd9n" Oct 30 21:59:46.139331 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:46.139287 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64d0f5bd-9b36-4ec4-b3c3-d47976e760e9-metrics-certs\") pod \"maas-api-e2e-ipp-b-19cab1-5855fdfb65-zxd9n\" (UID: \"64d0f5bd-9b36-4ec4-b3c3-d47976e760e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-19cab1-5855fdfb65-zxd9n" Oct 30 21:59:46.139429 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:46.139349 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64d0f5bd-9b36-4ec4-b3c3-d47976e760e9-maas-api-tls\") pod \"maas-api-e2e-ipp-b-19cab1-5855fdfb65-zxd9n\" (UID: \"64d0f5bd-9b36-4ec4-b3c3-d47976e760e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-19cab1-5855fdfb65-zxd9n" Oct 30 21:59:46.139897 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:46.139869 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64d0f5bd-9b36-4ec4-b3c3-d47976e760e9-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-19cab1-5855fdfb65-zxd9n\" (UID: \"64d0f5bd-9b36-4ec4-b3c3-d47976e760e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-19cab1-5855fdfb65-zxd9n" Oct 30 21:59:46.142127 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:46.142101 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64d0f5bd-9b36-4ec4-b3c3-d47976e760e9-metrics-certs\") pod \"maas-api-e2e-ipp-b-19cab1-5855fdfb65-zxd9n\" (UID: \"64d0f5bd-9b36-4ec4-b3c3-d47976e760e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-19cab1-5855fdfb65-zxd9n" Oct 30 21:59:46.142238 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:46.142134 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64d0f5bd-9b36-4ec4-b3c3-d47976e760e9-maas-api-tls\") pod \"maas-api-e2e-ipp-b-19cab1-5855fdfb65-zxd9n\" (UID: \"64d0f5bd-9b36-4ec4-b3c3-d47976e760e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-19cab1-5855fdfb65-zxd9n" Oct 30 21:59:46.182714 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:46.182666 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64sqf\" (UniqueName: \"kubernetes.io/projected/64d0f5bd-9b36-4ec4-b3c3-d47976e760e9-kube-api-access-64sqf\") pod \"maas-api-e2e-ipp-b-19cab1-5855fdfb65-zxd9n\" (UID: \"64d0f5bd-9b36-4ec4-b3c3-d47976e760e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-19cab1-5855fdfb65-zxd9n" Oct 30 21:59:46.229011 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:46.228980 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:46.335891 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:46.335814 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-19cab1-5855fdfb65-zxd9n" Oct 30 21:59:46.389685 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:46.389648 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:46.628728 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:46.628699 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-19cab1-5855fdfb65-zxd9n"] Oct 30 21:59:46.638737 ip-10-0-137-181 kubenswrapper[6720]: W1030 21:59:46.638710 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64d0f5bd_9b36_4ec4_b3c3_d47976e760e9.slice/crio-f2bd04dbdc10e443923f6dff056bcc66d80c17d4de163a15577ae602a86102f9 WatchSource:0}: Error finding container f2bd04dbdc10e443923f6dff056bcc66d80c17d4de163a15577ae602a86102f9: Status 404 returned error can't find the container with id f2bd04dbdc10e443923f6dff056bcc66d80c17d4de163a15577ae602a86102f9 Oct 30 21:59:47.115927 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:47.115885 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:47.214987 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:47.214945 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-19cab1-5855fdfb65-zxd9n" event={"ID":"64d0f5bd-9b36-4ec4-b3c3-d47976e760e9","Type":"ContainerStarted","Data":"f2bd04dbdc10e443923f6dff056bcc66d80c17d4de163a15577ae602a86102f9"} Oct 30 21:59:47.883999 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:47.883963 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:48.123405 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:48.120954 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:48.223452 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:48.223346 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-19cab1-5855fdfb65-zxd9n" event={"ID":"64d0f5bd-9b36-4ec4-b3c3-d47976e760e9","Type":"ContainerStarted","Data":"342e9e768371b05c5895868ce9e338201735f509054a29ba9063deafad5dbb68"} Oct 30 21:59:48.223628 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:48.223504 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-19cab1-5855fdfb65-zxd9n" Oct 30 21:59:48.248824 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:48.248749 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-19cab1-5855fdfb65-zxd9n" podStartSLOduration=2.741526513 podStartE2EDuration="3.248730519s" totalImagesPullingTime="507.204006ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:48.244669356 +0000 UTC m=+1154.415634539" watchObservedRunningTime="2026-10-30 21:59:48.248730519 +0000 UTC m=+1154.419695708" Oct 30 21:59:48.280269 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:48.280222 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:48.973209 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:48.973168 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:49.721826 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:49.721789 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:50.614951 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:50.614894 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:51.298870 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:51.298833 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:52.241573 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:52.241542 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:52.911114 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:52.911084 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:53.737882 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:53.737840 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:54.252671 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:54.252644 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-19cab1-5855fdfb65-zxd9n" Oct 30 21:59:54.349661 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:54.349614 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:54.769412 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:54.766987 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:55.003308 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:55.003271 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:56.111089 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:56.111060 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:56.792319 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:56.792288 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:57.627174 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:57.627119 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:57.982234 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:57.982149 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:58.265158 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:58.265061 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:58.584666 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:58.584570 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 21:59:59.342951 ip-10-0-137-181 kubenswrapper[6720]: I1030 21:59:59.342918 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:00.342002 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:00.340489 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889960-8545q"] Oct 30 22:00:00.347471 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:00.347443 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f47f79-29889960-g7ktn"] Oct 30 22:00:00.349065 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:00.349043 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889960-8545q" Oct 30 22:00:00.353895 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:00.353867 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:00.356545 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:00.356521 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f47f79-29889960-g7ktn" Oct 30 22:00:00.377104 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:00.377078 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f47f79-29889960-g7ktn"] Oct 30 22:00:00.380116 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:00.380078 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:00:00.380622 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:00.380598 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889960-8545q"] Oct 30 22:00:00.383322 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:00.383299 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-939a16-29889960-ngv6s"] Oct 30 22:00:00.399632 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:00.399554 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-939a16-29889960-ngv6s" Oct 30 22:00:00.404148 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:00.402925 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-939a16-29889960-ngv6s"] Oct 30 22:00:00.419148 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:00.419118 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae339224-a152-44ec-8c6e-9545fa9821a9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-f47f79-29889960-g7ktn\" (UID: \"ae339224-a152-44ec-8c6e-9545fa9821a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f47f79-29889960-g7ktn" Oct 30 22:00:00.419462 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:00.419436 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs2c6\" (UniqueName: \"kubernetes.io/projected/ae339224-a152-44ec-8c6e-9545fa9821a9-kube-api-access-hs2c6\") pod \"maas-api-key-cleanup-e2e-api-b-f47f79-29889960-g7ktn\" (UID: \"ae339224-a152-44ec-8c6e-9545fa9821a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f47f79-29889960-g7ktn" Oct 30 22:00:00.419584 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:00.419505 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2093daf-0e20-462a-be3f-c65e6a136333-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889960-8545q\" (UID: \"e2093daf-0e20-462a-be3f-c65e6a136333\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889960-8545q" Oct 30 22:00:00.419649 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:00.419595 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mrht\" (UniqueName: \"kubernetes.io/projected/e2093daf-0e20-462a-be3f-c65e6a136333-kube-api-access-2mrht\") pod \"maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889960-8545q\" (UID: \"e2093daf-0e20-462a-be3f-c65e6a136333\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889960-8545q" Oct 30 22:00:00.520942 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:00.520912 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs2c6\" (UniqueName: \"kubernetes.io/projected/ae339224-a152-44ec-8c6e-9545fa9821a9-kube-api-access-hs2c6\") pod \"maas-api-key-cleanup-e2e-api-b-f47f79-29889960-g7ktn\" (UID: \"ae339224-a152-44ec-8c6e-9545fa9821a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f47f79-29889960-g7ktn" Oct 30 22:00:00.521213 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:00.520948 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85zz6\" (UniqueName: \"kubernetes.io/projected/e04efe02-4bd6-48bf-b953-0ee356f2bc88-kube-api-access-85zz6\") pod \"maas-api-key-cleanup-e2e-worker-w4-939a16-29889960-ngv6s\" (UID: \"e04efe02-4bd6-48bf-b953-0ee356f2bc88\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-939a16-29889960-ngv6s" Oct 30 22:00:00.521213 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:00.520982 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2093daf-0e20-462a-be3f-c65e6a136333-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889960-8545q\" (UID: \"e2093daf-0e20-462a-be3f-c65e6a136333\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889960-8545q" Oct 30 22:00:00.521213 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:00.521063 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mrht\" (UniqueName: \"kubernetes.io/projected/e2093daf-0e20-462a-be3f-c65e6a136333-kube-api-access-2mrht\") pod \"maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889960-8545q\" (UID: \"e2093daf-0e20-462a-be3f-c65e6a136333\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889960-8545q" Oct 30 22:00:00.521213 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:00.521171 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae339224-a152-44ec-8c6e-9545fa9821a9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-f47f79-29889960-g7ktn\" (UID: \"ae339224-a152-44ec-8c6e-9545fa9821a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f47f79-29889960-g7ktn" Oct 30 22:00:00.521501 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:00.521243 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e04efe02-4bd6-48bf-b953-0ee356f2bc88-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-939a16-29889960-ngv6s\" (UID: \"e04efe02-4bd6-48bf-b953-0ee356f2bc88\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-939a16-29889960-ngv6s" Oct 30 22:00:00.521754 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:00.521730 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2093daf-0e20-462a-be3f-c65e6a136333-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889960-8545q\" (UID: \"e2093daf-0e20-462a-be3f-c65e6a136333\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889960-8545q" Oct 30 22:00:00.521982 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:00.521957 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae339224-a152-44ec-8c6e-9545fa9821a9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-f47f79-29889960-g7ktn\" (UID: \"ae339224-a152-44ec-8c6e-9545fa9821a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f47f79-29889960-g7ktn" Oct 30 22:00:00.546417 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:00.546370 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs2c6\" (UniqueName: \"kubernetes.io/projected/ae339224-a152-44ec-8c6e-9545fa9821a9-kube-api-access-hs2c6\") pod \"maas-api-key-cleanup-e2e-api-b-f47f79-29889960-g7ktn\" (UID: \"ae339224-a152-44ec-8c6e-9545fa9821a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f47f79-29889960-g7ktn" Oct 30 22:00:00.550664 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:00.550642 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mrht\" (UniqueName: \"kubernetes.io/projected/e2093daf-0e20-462a-be3f-c65e6a136333-kube-api-access-2mrht\") pod \"maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889960-8545q\" (UID: \"e2093daf-0e20-462a-be3f-c65e6a136333\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889960-8545q" Oct 30 22:00:00.622753 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:00.622718 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e04efe02-4bd6-48bf-b953-0ee356f2bc88-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-939a16-29889960-ngv6s\" (UID: \"e04efe02-4bd6-48bf-b953-0ee356f2bc88\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-939a16-29889960-ngv6s" Oct 30 22:00:00.622938 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:00.622857 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85zz6\" (UniqueName: \"kubernetes.io/projected/e04efe02-4bd6-48bf-b953-0ee356f2bc88-kube-api-access-85zz6\") pod \"maas-api-key-cleanup-e2e-worker-w4-939a16-29889960-ngv6s\" (UID: \"e04efe02-4bd6-48bf-b953-0ee356f2bc88\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-939a16-29889960-ngv6s" Oct 30 22:00:00.623369 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:00.623351 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e04efe02-4bd6-48bf-b953-0ee356f2bc88-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-939a16-29889960-ngv6s\" (UID: \"e04efe02-4bd6-48bf-b953-0ee356f2bc88\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-939a16-29889960-ngv6s" Oct 30 22:00:00.637061 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:00.637027 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85zz6\" (UniqueName: \"kubernetes.io/projected/e04efe02-4bd6-48bf-b953-0ee356f2bc88-kube-api-access-85zz6\") pod \"maas-api-key-cleanup-e2e-worker-w4-939a16-29889960-ngv6s\" (UID: \"e04efe02-4bd6-48bf-b953-0ee356f2bc88\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-939a16-29889960-ngv6s" Oct 30 22:00:00.688921 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:00.688888 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889960-8545q" Oct 30 22:00:00.705284 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:00.705256 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f47f79-29889960-g7ktn" Oct 30 22:00:00.718511 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:00.718483 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-939a16-29889960-ngv6s" Oct 30 22:00:01.000474 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:01.000439 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889960-8545q"] Oct 30 22:00:01.001104 ip-10-0-137-181 kubenswrapper[6720]: W1030 22:00:01.001084 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2093daf_0e20_462a_be3f_c65e6a136333.slice/crio-8cf1b8aa668683633a44f13530a955d3f84af0d9ec912be381e91ddae96e79d0 WatchSource:0}: Error finding container 8cf1b8aa668683633a44f13530a955d3f84af0d9ec912be381e91ddae96e79d0: Status 404 returned error can't find the container with id 8cf1b8aa668683633a44f13530a955d3f84af0d9ec912be381e91ddae96e79d0 Oct 30 22:00:01.019741 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:01.019718 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f47f79-29889960-g7ktn"] Oct 30 22:00:01.020634 ip-10-0-137-181 kubenswrapper[6720]: W1030 22:00:01.020610 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae339224_a152_44ec_8c6e_9545fa9821a9.slice/crio-108cbc6bc598bc8f23f0daa6a53e86e7838ff97a5fc1203a6f2aca19c6fad63f WatchSource:0}: Error finding container 108cbc6bc598bc8f23f0daa6a53e86e7838ff97a5fc1203a6f2aca19c6fad63f: Status 404 returned error can't find the container with id 108cbc6bc598bc8f23f0daa6a53e86e7838ff97a5fc1203a6f2aca19c6fad63f Oct 30 22:00:01.022874 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:01.022837 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-939a16-29889960-ngv6s"] Oct 30 22:00:01.067037 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:01.066993 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:01.373450 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:01.368527 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889960-8545q" event={"ID":"e2093daf-0e20-462a-be3f-c65e6a136333","Type":"ContainerStarted","Data":"8cf1b8aa668683633a44f13530a955d3f84af0d9ec912be381e91ddae96e79d0"} Oct 30 22:00:01.373450 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:01.370673 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-939a16-29889960-ngv6s" event={"ID":"e04efe02-4bd6-48bf-b953-0ee356f2bc88","Type":"ContainerStarted","Data":"6592f7759b7184fa326ac9e3855ef16b2516714865b9c0ebe2414a9dce6858f7"} Oct 30 22:00:01.373450 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:01.372494 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f47f79-29889960-g7ktn" event={"ID":"ae339224-a152-44ec-8c6e-9545fa9821a9","Type":"ContainerStarted","Data":"108cbc6bc598bc8f23f0daa6a53e86e7838ff97a5fc1203a6f2aca19c6fad63f"} Oct 30 22:00:01.527039 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:01.526951 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:01.792205 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:01.792116 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:02.383191 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:02.383152 6720 generic.go:356] "Generic (PLEG): container finished" podID="e2093daf-0e20-462a-be3f-c65e6a136333" containerID="f1f632ee40318968b591d8700449bc8d79cf34e7b85d766abfb862809844ba78" exitCode=60 Oct 30 22:00:02.383721 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:02.383238 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889960-8545q" event={"ID":"e2093daf-0e20-462a-be3f-c65e6a136333","Type":"ContainerDied","Data":"f1f632ee40318968b591d8700449bc8d79cf34e7b85d766abfb862809844ba78"} Oct 30 22:00:02.383721 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:02.383665 6720 scope.go:122] "RemoveContainer" containerID="f1f632ee40318968b591d8700449bc8d79cf34e7b85d766abfb862809844ba78" Oct 30 22:00:02.385087 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:02.385064 6720 generic.go:356] "Generic (PLEG): container finished" podID="e04efe02-4bd6-48bf-b953-0ee356f2bc88" containerID="2142236adf6c9609b924557d2d610779ed92e4b5e658b3e99952800ff26104a7" exitCode=60 Oct 30 22:00:02.385201 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:02.385148 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-939a16-29889960-ngv6s" event={"ID":"e04efe02-4bd6-48bf-b953-0ee356f2bc88","Type":"ContainerDied","Data":"2142236adf6c9609b924557d2d610779ed92e4b5e658b3e99952800ff26104a7"} Oct 30 22:00:02.385468 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:02.385452 6720 scope.go:122] "RemoveContainer" containerID="2142236adf6c9609b924557d2d610779ed92e4b5e658b3e99952800ff26104a7" Oct 30 22:00:02.387157 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:02.387063 6720 generic.go:356] "Generic (PLEG): container finished" podID="ae339224-a152-44ec-8c6e-9545fa9821a9" containerID="660d60debb3401df526bd7d6c68ebab51359dbdcf7aade4ba54666a27ffbb49c" exitCode=60 Oct 30 22:00:02.387157 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:02.387099 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f47f79-29889960-g7ktn" event={"ID":"ae339224-a152-44ec-8c6e-9545fa9821a9","Type":"ContainerDied","Data":"660d60debb3401df526bd7d6c68ebab51359dbdcf7aade4ba54666a27ffbb49c"} Oct 30 22:00:02.387326 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:02.387312 6720 scope.go:122] "RemoveContainer" containerID="660d60debb3401df526bd7d6c68ebab51359dbdcf7aade4ba54666a27ffbb49c" Oct 30 22:00:02.783857 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:02.783833 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:03.056439 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:03.056328 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:03.396440 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:03.396369 6720 generic.go:356] "Generic (PLEG): container finished" podID="ae339224-a152-44ec-8c6e-9545fa9821a9" containerID="95a0f88a923fcdacc1063b8e32b16248ba771bb8489b9173b3c482a789249a30" exitCode=60 Oct 30 22:00:03.396836 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:03.396459 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f47f79-29889960-g7ktn" event={"ID":"ae339224-a152-44ec-8c6e-9545fa9821a9","Type":"ContainerDied","Data":"95a0f88a923fcdacc1063b8e32b16248ba771bb8489b9173b3c482a789249a30"} Oct 30 22:00:03.396836 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:03.396536 6720 scope.go:122] "RemoveContainer" containerID="660d60debb3401df526bd7d6c68ebab51359dbdcf7aade4ba54666a27ffbb49c" Oct 30 22:00:03.396918 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:03.396840 6720 scope.go:122] "RemoveContainer" containerID="95a0f88a923fcdacc1063b8e32b16248ba771bb8489b9173b3c482a789249a30" Oct 30 22:00:03.397231 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:00:03.397204 6720 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-f47f79-29889960-g7ktn_odh-ai-gateway-infra(ae339224-a152-44ec-8c6e-9545fa9821a9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f47f79-29889960-g7ktn" podUID="ae339224-a152-44ec-8c6e-9545fa9821a9" Oct 30 22:00:03.398959 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:03.398940 6720 generic.go:356] "Generic (PLEG): container finished" podID="e2093daf-0e20-462a-be3f-c65e6a136333" containerID="86d8a895b72a7a38586e055199c18ba288c64841199104dc4def81bc26623b87" exitCode=60 Oct 30 22:00:03.399061 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:03.398968 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889960-8545q" event={"ID":"e2093daf-0e20-462a-be3f-c65e6a136333","Type":"ContainerDied","Data":"86d8a895b72a7a38586e055199c18ba288c64841199104dc4def81bc26623b87"} Oct 30 22:00:03.399273 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:03.399257 6720 scope.go:122] "RemoveContainer" containerID="86d8a895b72a7a38586e055199c18ba288c64841199104dc4def81bc26623b87" Oct 30 22:00:03.399667 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:00:03.399647 6720 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889960-8545q_odh-ai-gateway-infra(e2093daf-0e20-462a-be3f-c65e6a136333)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889960-8545q" podUID="e2093daf-0e20-462a-be3f-c65e6a136333" Oct 30 22:00:03.400875 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:03.400845 6720 generic.go:356] "Generic (PLEG): container finished" podID="e04efe02-4bd6-48bf-b953-0ee356f2bc88" containerID="40038aba06d393296be85413db85e9aea7ba10fa7a7fc0d01e759b3a43dc237c" exitCode=60 Oct 30 22:00:03.400978 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:03.400963 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-939a16-29889960-ngv6s" event={"ID":"e04efe02-4bd6-48bf-b953-0ee356f2bc88","Type":"ContainerDied","Data":"40038aba06d393296be85413db85e9aea7ba10fa7a7fc0d01e759b3a43dc237c"} Oct 30 22:00:03.401200 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:03.401157 6720 scope.go:122] "RemoveContainer" containerID="40038aba06d393296be85413db85e9aea7ba10fa7a7fc0d01e759b3a43dc237c" Oct 30 22:00:03.401425 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:00:03.401403 6720 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-939a16-29889960-ngv6s_odh-ai-gateway-infra(e04efe02-4bd6-48bf-b953-0ee356f2bc88)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-939a16-29889960-ngv6s" podUID="e04efe02-4bd6-48bf-b953-0ee356f2bc88" Oct 30 22:00:03.557539 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:03.557509 6720 scope.go:122] "RemoveContainer" containerID="f1f632ee40318968b591d8700449bc8d79cf34e7b85d766abfb862809844ba78" Oct 30 22:00:03.570844 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:03.570825 6720 scope.go:122] "RemoveContainer" containerID="2142236adf6c9609b924557d2d610779ed92e4b5e658b3e99952800ff26104a7" Oct 30 22:00:04.226530 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:04.226498 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:04.421593 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:04.421567 6720 scope.go:122] "RemoveContainer" containerID="95a0f88a923fcdacc1063b8e32b16248ba771bb8489b9173b3c482a789249a30" Oct 30 22:00:04.434411 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:00:04.433877 6720 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-f47f79-29889960-g7ktn_odh-ai-gateway-infra(ae339224-a152-44ec-8c6e-9545fa9821a9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f47f79-29889960-g7ktn" podUID="ae339224-a152-44ec-8c6e-9545fa9821a9" Oct 30 22:00:04.434411 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:04.433963 6720 scope.go:122] "RemoveContainer" containerID="86d8a895b72a7a38586e055199c18ba288c64841199104dc4def81bc26623b87" Oct 30 22:00:04.434411 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:00:04.434303 6720 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889960-8545q_odh-ai-gateway-infra(e2093daf-0e20-462a-be3f-c65e6a136333)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889960-8545q" podUID="e2093daf-0e20-462a-be3f-c65e6a136333" Oct 30 22:00:04.440221 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:04.440198 6720 scope.go:122] "RemoveContainer" containerID="40038aba06d393296be85413db85e9aea7ba10fa7a7fc0d01e759b3a43dc237c" Oct 30 22:00:04.440761 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:00:04.440731 6720 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-939a16-29889960-ngv6s_odh-ai-gateway-infra(e04efe02-4bd6-48bf-b953-0ee356f2bc88)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-939a16-29889960-ngv6s" podUID="e04efe02-4bd6-48bf-b953-0ee356f2bc88" Oct 30 22:00:04.470772 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:04.470733 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:05.627439 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:05.627402 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:06.449499 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:06.449466 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:07.244183 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:07.244151 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:08.003831 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:08.003796 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f47f79-29889960-g7ktn"] Oct 30 22:00:08.020605 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:08.020557 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:08.301461 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:08.298317 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:08.359605 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:08.359578 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f47f79-29889960-g7ktn" Oct 30 22:00:08.460139 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:08.460104 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae339224-a152-44ec-8c6e-9545fa9821a9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae339224-a152-44ec-8c6e-9545fa9821a9-maas-api-service-ca\") pod \"ae339224-a152-44ec-8c6e-9545fa9821a9\" (UID: \"ae339224-a152-44ec-8c6e-9545fa9821a9\") " Oct 30 22:00:08.460328 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:08.460157 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae339224-a152-44ec-8c6e-9545fa9821a9-kube-api-access-hs2c6\" (UniqueName: \"kubernetes.io/projected/ae339224-a152-44ec-8c6e-9545fa9821a9-kube-api-access-hs2c6\") pod \"ae339224-a152-44ec-8c6e-9545fa9821a9\" (UID: \"ae339224-a152-44ec-8c6e-9545fa9821a9\") " Oct 30 22:00:08.460663 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:08.460634 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae339224-a152-44ec-8c6e-9545fa9821a9-maas-api-service-ca" pod "ae339224-a152-44ec-8c6e-9545fa9821a9" (UID: "ae339224-a152-44ec-8c6e-9545fa9821a9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:00:08.461716 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:08.461690 6720 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae339224-a152-44ec-8c6e-9545fa9821a9-maas-api-service-ca\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:08.463528 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:08.463501 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae339224-a152-44ec-8c6e-9545fa9821a9-kube-api-access-hs2c6" pod "ae339224-a152-44ec-8c6e-9545fa9821a9" (UID: "ae339224-a152-44ec-8c6e-9545fa9821a9"). InnerVolumeSpecName "kube-api-access-hs2c6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:08.469543 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:08.469515 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f47f79-29889960-g7ktn" event={"ID":"ae339224-a152-44ec-8c6e-9545fa9821a9","Type":"ContainerDied","Data":"108cbc6bc598bc8f23f0daa6a53e86e7838ff97a5fc1203a6f2aca19c6fad63f"} Oct 30 22:00:08.469640 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:08.469582 6720 scope.go:122] "RemoveContainer" containerID="95a0f88a923fcdacc1063b8e32b16248ba771bb8489b9173b3c482a789249a30" Oct 30 22:00:08.469640 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:08.469526 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f47f79-29889960-g7ktn" Oct 30 22:00:08.502911 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:08.502881 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f47f79-29889960-g7ktn"] Oct 30 22:00:08.506260 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:08.506231 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-f47f79-29889960-g7ktn"] Oct 30 22:00:08.513332 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:08.513297 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:08.563027 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:08.562950 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hs2c6\" (UniqueName: \"kubernetes.io/projected/ae339224-a152-44ec-8c6e-9545fa9821a9-kube-api-access-hs2c6\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:08.959083 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:08.959035 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:09.117223 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:09.117194 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:09.501705 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:09.501664 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:09.757953 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:09.755180 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:09.958625 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:09.958592 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae339224-a152-44ec-8c6e-9545fa9821a9" path="/var/lib/kubelet/pods/ae339224-a152-44ec-8c6e-9545fa9821a9/volumes" Oct 30 22:00:09.959340 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:09.959310 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:10.408686 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:00:10.408648 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 22:00:10.503215 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:10.503180 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:11.281322 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:11.281287 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:11.987163 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:11.987125 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:12.181292 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:12.181252 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:12.837616 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:12.837583 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:13.741277 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:13.741193 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:14.429530 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:14.429500 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:14.950034 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:14.949997 6720 scope.go:122] "RemoveContainer" containerID="40038aba06d393296be85413db85e9aea7ba10fa7a7fc0d01e759b3a43dc237c" Oct 30 22:00:15.137002 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:15.136970 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:15.330080 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:15.330015 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:15.567246 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:15.567212 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-939a16-29889960-ngv6s" event={"ID":"e04efe02-4bd6-48bf-b953-0ee356f2bc88","Type":"ContainerStarted","Data":"a4cf59b99e59995380fec32b83c8409380dd95bf9aba7fb59d1e535bd2ab1f41"} Oct 30 22:00:15.608703 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:15.608651 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-939a16-29889960-ngv6s" podStartSLOduration=15.608637599 podStartE2EDuration="15.608637599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:15.605510691 +0000 UTC m=+1181.776475877" watchObservedRunningTime="2026-10-30 22:00:15.608637599 +0000 UTC m=+1181.779602844" Oct 30 22:00:16.286522 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:16.286434 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:16.467011 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:16.466977 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:16.575737 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:16.575655 6720 generic.go:356] "Generic (PLEG): container finished" podID="e04efe02-4bd6-48bf-b953-0ee356f2bc88" containerID="a4cf59b99e59995380fec32b83c8409380dd95bf9aba7fb59d1e535bd2ab1f41" exitCode=60 Oct 30 22:00:16.575737 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:16.575701 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-939a16-29889960-ngv6s" event={"ID":"e04efe02-4bd6-48bf-b953-0ee356f2bc88","Type":"ContainerDied","Data":"a4cf59b99e59995380fec32b83c8409380dd95bf9aba7fb59d1e535bd2ab1f41"} Oct 30 22:00:16.575969 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:16.575754 6720 scope.go:122] "RemoveContainer" containerID="40038aba06d393296be85413db85e9aea7ba10fa7a7fc0d01e759b3a43dc237c" Oct 30 22:00:16.576233 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:16.576204 6720 scope.go:122] "RemoveContainer" containerID="a4cf59b99e59995380fec32b83c8409380dd95bf9aba7fb59d1e535bd2ab1f41" Oct 30 22:00:16.576629 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:00:16.576591 6720 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-939a16-29889960-ngv6s_odh-ai-gateway-infra(e04efe02-4bd6-48bf-b953-0ee356f2bc88)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-939a16-29889960-ngv6s" podUID="e04efe02-4bd6-48bf-b953-0ee356f2bc88" Oct 30 22:00:16.621242 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:16.621211 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-939a16-29889960-ngv6s"] Oct 30 22:00:16.823645 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:16.823612 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:17.143999 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:17.143971 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:17.808932 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:17.808903 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-939a16-29889960-ngv6s" Oct 30 22:00:17.949364 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:17.949321 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:17.952254 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:17.952232 6720 scope.go:122] "RemoveContainer" containerID="86d8a895b72a7a38586e055199c18ba288c64841199104dc4def81bc26623b87" Oct 30 22:00:17.962580 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:17.962533 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e04efe02-4bd6-48bf-b953-0ee356f2bc88-kube-api-access-85zz6" pod "e04efe02-4bd6-48bf-b953-0ee356f2bc88" (UID: "e04efe02-4bd6-48bf-b953-0ee356f2bc88"). InnerVolumeSpecName "kube-api-access-85zz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:17.962580 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:17.959022 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e04efe02-4bd6-48bf-b953-0ee356f2bc88-kube-api-access-85zz6\" (UniqueName: \"kubernetes.io/projected/e04efe02-4bd6-48bf-b953-0ee356f2bc88-kube-api-access-85zz6\") pod \"e04efe02-4bd6-48bf-b953-0ee356f2bc88\" (UID: \"e04efe02-4bd6-48bf-b953-0ee356f2bc88\") " Oct 30 22:00:17.962945 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:17.962875 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e04efe02-4bd6-48bf-b953-0ee356f2bc88-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e04efe02-4bd6-48bf-b953-0ee356f2bc88-maas-api-service-ca\") pod \"e04efe02-4bd6-48bf-b953-0ee356f2bc88\" (UID: \"e04efe02-4bd6-48bf-b953-0ee356f2bc88\") " Oct 30 22:00:17.963571 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:17.963548 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85zz6\" (UniqueName: \"kubernetes.io/projected/e04efe02-4bd6-48bf-b953-0ee356f2bc88-kube-api-access-85zz6\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:17.963843 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:17.963800 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e04efe02-4bd6-48bf-b953-0ee356f2bc88-maas-api-service-ca" pod "e04efe02-4bd6-48bf-b953-0ee356f2bc88" (UID: "e04efe02-4bd6-48bf-b953-0ee356f2bc88"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:00:18.065367 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:18.065332 6720 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e04efe02-4bd6-48bf-b953-0ee356f2bc88-maas-api-service-ca\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:18.595833 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:18.595749 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889960-8545q" event={"ID":"e2093daf-0e20-462a-be3f-c65e6a136333","Type":"ContainerStarted","Data":"a5860ddc2d3f513ebd98d3d02628ecfead9f93e9cf54c9c77c5b15c32313f339"} Oct 30 22:00:18.597275 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:18.597254 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-939a16-29889960-ngv6s" Oct 30 22:00:18.597504 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:18.597278 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-939a16-29889960-ngv6s" event={"ID":"e04efe02-4bd6-48bf-b953-0ee356f2bc88","Type":"ContainerDied","Data":"6592f7759b7184fa326ac9e3855ef16b2516714865b9c0ebe2414a9dce6858f7"} Oct 30 22:00:18.597504 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:18.597324 6720 scope.go:122] "RemoveContainer" containerID="a4cf59b99e59995380fec32b83c8409380dd95bf9aba7fb59d1e535bd2ab1f41" Oct 30 22:00:18.616513 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:18.616467 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889960-8545q" podStartSLOduration=18.616375586 podStartE2EDuration="18.616375586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:18.615474093 +0000 UTC m=+1184.786439275" watchObservedRunningTime="2026-10-30 22:00:18.616375586 +0000 UTC m=+1184.787340814" Oct 30 22:00:18.635419 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:18.635360 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:18.644086 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:18.644059 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-939a16-29889960-ngv6s"] Oct 30 22:00:18.660141 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:18.660116 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-939a16-29889960-ngv6s"] Oct 30 22:00:18.828590 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:18.828558 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:19.044331 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:19.044295 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:19.581119 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:19.581078 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:19.606051 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:19.606021 6720 generic.go:356] "Generic (PLEG): container finished" podID="e2093daf-0e20-462a-be3f-c65e6a136333" containerID="a5860ddc2d3f513ebd98d3d02628ecfead9f93e9cf54c9c77c5b15c32313f339" exitCode=60 Oct 30 22:00:19.606233 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:19.606090 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889960-8545q" event={"ID":"e2093daf-0e20-462a-be3f-c65e6a136333","Type":"ContainerDied","Data":"a5860ddc2d3f513ebd98d3d02628ecfead9f93e9cf54c9c77c5b15c32313f339"} Oct 30 22:00:19.606233 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:19.606163 6720 scope.go:122] "RemoveContainer" containerID="86d8a895b72a7a38586e055199c18ba288c64841199104dc4def81bc26623b87" Oct 30 22:00:19.606614 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:19.606589 6720 scope.go:122] "RemoveContainer" containerID="a5860ddc2d3f513ebd98d3d02628ecfead9f93e9cf54c9c77c5b15c32313f339" Oct 30 22:00:19.606920 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:00:19.606885 6720 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889960-8545q_odh-ai-gateway-infra(e2093daf-0e20-462a-be3f-c65e6a136333)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889960-8545q" podUID="e2093daf-0e20-462a-be3f-c65e6a136333" Oct 30 22:00:19.635129 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:19.635101 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889960-8545q"] Oct 30 22:00:19.954985 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:19.954858 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:19.961075 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:19.961030 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e04efe02-4bd6-48bf-b953-0ee356f2bc88" path="/var/lib/kubelet/pods/e04efe02-4bd6-48bf-b953-0ee356f2bc88/volumes" Oct 30 22:00:20.120132 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:20.120097 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:20.305891 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:20.305803 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:20.807379 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:20.807355 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889960-8545q" Oct 30 22:00:20.919971 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:20.919887 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2093daf-0e20-462a-be3f-c65e6a136333-kube-api-access-2mrht\" (UniqueName: \"kubernetes.io/projected/e2093daf-0e20-462a-be3f-c65e6a136333-kube-api-access-2mrht\") pod \"e2093daf-0e20-462a-be3f-c65e6a136333\" (UID: \"e2093daf-0e20-462a-be3f-c65e6a136333\") " Oct 30 22:00:20.920722 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:20.920697 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2093daf-0e20-462a-be3f-c65e6a136333-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2093daf-0e20-462a-be3f-c65e6a136333-maas-api-service-ca\") pod \"e2093daf-0e20-462a-be3f-c65e6a136333\" (UID: \"e2093daf-0e20-462a-be3f-c65e6a136333\") " Oct 30 22:00:20.921908 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:20.921192 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2093daf-0e20-462a-be3f-c65e6a136333-maas-api-service-ca" pod "e2093daf-0e20-462a-be3f-c65e6a136333" (UID: "e2093daf-0e20-462a-be3f-c65e6a136333"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:00:20.924146 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:20.924119 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2093daf-0e20-462a-be3f-c65e6a136333-kube-api-access-2mrht" pod "e2093daf-0e20-462a-be3f-c65e6a136333" (UID: "e2093daf-0e20-462a-be3f-c65e6a136333"). InnerVolumeSpecName "kube-api-access-2mrht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:21.023735 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:21.023702 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mrht\" (UniqueName: \"kubernetes.io/projected/e2093daf-0e20-462a-be3f-c65e6a136333-kube-api-access-2mrht\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:21.023735 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:21.023732 6720 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2093daf-0e20-462a-be3f-c65e6a136333-maas-api-service-ca\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:21.049872 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:21.049830 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:21.647882 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:21.647851 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889960-8545q" Oct 30 22:00:21.647882 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:21.647860 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889960-8545q" event={"ID":"e2093daf-0e20-462a-be3f-c65e6a136333","Type":"ContainerDied","Data":"8cf1b8aa668683633a44f13530a955d3f84af0d9ec912be381e91ddae96e79d0"} Oct 30 22:00:21.648096 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:21.647960 6720 scope.go:122] "RemoveContainer" containerID="a5860ddc2d3f513ebd98d3d02628ecfead9f93e9cf54c9c77c5b15c32313f339" Oct 30 22:00:21.678325 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:21.678299 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889960-8545q"] Oct 30 22:00:21.680912 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:21.680892 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-0f26c5-29889960-8545q"] Oct 30 22:00:21.957849 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:21.957757 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2093daf-0e20-462a-be3f-c65e6a136333" path="/var/lib/kubelet/pods/e2093daf-0e20-462a-be3f-c65e6a136333/volumes" Oct 30 22:00:22.243449 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:22.243353 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:22.791088 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:22.791058 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:22.961484 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:22.961453 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:23.329397 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:23.329352 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:23.860023 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:23.859932 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:24.038184 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:24.038144 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:24.225355 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:24.225263 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:25.153159 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:25.153123 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:25.556442 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:25.556293 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:25.741967 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:25.741930 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:26.124304 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:26.124271 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:26.908373 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:26.908312 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:27.066924 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:27.066881 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:27.883311 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:27.883266 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:28.504241 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:28.504209 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:29.503077 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:29.502874 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:29.881012 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:29.880977 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:30.044638 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:30.044597 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:30.601811 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:30.601778 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:30.603261 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:30.603216 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-gxcgh"] Oct 30 22:00:30.604662 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:30.604625 6720 cpu_manager.go:414] "containerMap: removing container" podUID="e04efe02-4bd6-48bf-b953-0ee356f2bc88" containerName="cleanup" Oct 30 22:00:30.604798 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:30.604783 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e04efe02-4bd6-48bf-b953-0ee356f2bc88" containerName="cleanup" Oct 30 22:00:30.604916 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:30.604906 6720 cpu_manager.go:414] "containerMap: removing container" podUID="ae339224-a152-44ec-8c6e-9545fa9821a9" containerName="cleanup" Oct 30 22:00:30.604979 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:30.604968 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae339224-a152-44ec-8c6e-9545fa9821a9" containerName="cleanup" Oct 30 22:00:30.605082 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:30.605072 6720 cpu_manager.go:414] "containerMap: removing container" podUID="e2093daf-0e20-462a-be3f-c65e6a136333" containerName="cleanup" Oct 30 22:00:30.605161 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:30.605150 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2093daf-0e20-462a-be3f-c65e6a136333" containerName="cleanup" Oct 30 22:00:30.605232 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:30.605222 6720 cpu_manager.go:414] "containerMap: removing container" podUID="ae339224-a152-44ec-8c6e-9545fa9821a9" containerName="cleanup" Oct 30 22:00:30.605310 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:30.605297 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae339224-a152-44ec-8c6e-9545fa9821a9" containerName="cleanup" Oct 30 22:00:30.605419 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:30.605408 6720 cpu_manager.go:414] "containerMap: removing container" podUID="e2093daf-0e20-462a-be3f-c65e6a136333" containerName="cleanup" Oct 30 22:00:30.605492 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:30.605483 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2093daf-0e20-462a-be3f-c65e6a136333" containerName="cleanup" Oct 30 22:00:30.605584 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:30.605575 6720 cpu_manager.go:414] "containerMap: removing container" podUID="e04efe02-4bd6-48bf-b953-0ee356f2bc88" containerName="cleanup" Oct 30 22:00:30.605655 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:30.605646 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e04efe02-4bd6-48bf-b953-0ee356f2bc88" containerName="cleanup" Oct 30 22:00:30.605905 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:30.605854 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="e04efe02-4bd6-48bf-b953-0ee356f2bc88" containerName="cleanup" Oct 30 22:00:30.606016 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:30.606005 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae339224-a152-44ec-8c6e-9545fa9821a9" containerName="cleanup" Oct 30 22:00:30.606091 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:30.606083 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2093daf-0e20-462a-be3f-c65e6a136333" containerName="cleanup" Oct 30 22:00:30.606163 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:30.606154 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae339224-a152-44ec-8c6e-9545fa9821a9" containerName="cleanup" Oct 30 22:00:30.606244 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:30.606235 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2093daf-0e20-462a-be3f-c65e6a136333" containerName="cleanup" Oct 30 22:00:30.606311 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:30.606302 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="e04efe02-4bd6-48bf-b953-0ee356f2bc88" containerName="cleanup" Oct 30 22:00:30.612143 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:30.612120 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-gxcgh" Oct 30 22:00:30.619406 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:30.619363 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-3654d5-openshift-default-dockercfg-h757h\"" Oct 30 22:00:30.639875 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:30.639827 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-gxcgh"] Oct 30 22:00:30.706207 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:30.698917 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c73aea8-611d-4762-b2d2-6528eeb7dce9-istio-podinfo\") pod \"e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-gxcgh\" (UID: \"0c73aea8-611d-4762-b2d2-6528eeb7dce9\") " pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-gxcgh" Oct 30 22:00:30.706531 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:30.706488 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c73aea8-611d-4762-b2d2-6528eeb7dce9-workload-certs\") pod \"e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-gxcgh\" (UID: \"0c73aea8-611d-4762-b2d2-6528eeb7dce9\") " pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-gxcgh" Oct 30 22:00:30.706710 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:30.706554 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c73aea8-611d-4762-b2d2-6528eeb7dce9-credential-socket\") pod \"e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-gxcgh\" (UID: \"0c73aea8-611d-4762-b2d2-6528eeb7dce9\") " pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-gxcgh" Oct 30 22:00:30.706710 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:30.706592 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c73aea8-611d-4762-b2d2-6528eeb7dce9-istio-token\") pod \"e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-gxcgh\" (UID: \"0c73aea8-611d-4762-b2d2-6528eeb7dce9\") " pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-gxcgh" Oct 30 22:00:30.706710 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:30.706622 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5j29\" (UniqueName: \"kubernetes.io/projected/0c73aea8-611d-4762-b2d2-6528eeb7dce9-kube-api-access-d5j29\") pod \"e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-gxcgh\" (UID: \"0c73aea8-611d-4762-b2d2-6528eeb7dce9\") " pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-gxcgh" Oct 30 22:00:30.706891 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:30.706816 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c73aea8-611d-4762-b2d2-6528eeb7dce9-istio-data\") pod \"e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-gxcgh\" (UID: \"0c73aea8-611d-4762-b2d2-6528eeb7dce9\") " pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-gxcgh" Oct 30 22:00:30.706958 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:30.706941 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c73aea8-611d-4762-b2d2-6528eeb7dce9-istio-envoy\") pod \"e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-gxcgh\" (UID: \"0c73aea8-611d-4762-b2d2-6528eeb7dce9\") " pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-gxcgh" Oct 30 22:00:30.707177 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:30.707161 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c73aea8-611d-4762-b2d2-6528eeb7dce9-istiod-ca-cert\") pod \"e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-gxcgh\" (UID: \"0c73aea8-611d-4762-b2d2-6528eeb7dce9\") " pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-gxcgh" Oct 30 22:00:30.707306 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:30.707240 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c73aea8-611d-4762-b2d2-6528eeb7dce9-workload-socket\") pod \"e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-gxcgh\" (UID: \"0c73aea8-611d-4762-b2d2-6528eeb7dce9\") " pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-gxcgh" Oct 30 22:00:30.808849 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:30.808821 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c73aea8-611d-4762-b2d2-6528eeb7dce9-istio-podinfo\") pod \"e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-gxcgh\" (UID: \"0c73aea8-611d-4762-b2d2-6528eeb7dce9\") " pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-gxcgh" Oct 30 22:00:30.809021 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:30.808894 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c73aea8-611d-4762-b2d2-6528eeb7dce9-workload-certs\") pod \"e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-gxcgh\" (UID: \"0c73aea8-611d-4762-b2d2-6528eeb7dce9\") " pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-gxcgh" Oct 30 22:00:30.809021 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:30.808954 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c73aea8-611d-4762-b2d2-6528eeb7dce9-credential-socket\") pod \"e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-gxcgh\" (UID: \"0c73aea8-611d-4762-b2d2-6528eeb7dce9\") " pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-gxcgh" Oct 30 22:00:30.809021 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:30.808987 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c73aea8-611d-4762-b2d2-6528eeb7dce9-istio-token\") pod \"e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-gxcgh\" (UID: \"0c73aea8-611d-4762-b2d2-6528eeb7dce9\") " pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-gxcgh" Oct 30 22:00:30.809021 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:30.809013 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5j29\" (UniqueName: \"kubernetes.io/projected/0c73aea8-611d-4762-b2d2-6528eeb7dce9-kube-api-access-d5j29\") pod \"e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-gxcgh\" (UID: \"0c73aea8-611d-4762-b2d2-6528eeb7dce9\") " pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-gxcgh" Oct 30 22:00:30.809256 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:30.809097 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c73aea8-611d-4762-b2d2-6528eeb7dce9-istio-data\") pod \"e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-gxcgh\" (UID: \"0c73aea8-611d-4762-b2d2-6528eeb7dce9\") " pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-gxcgh" Oct 30 22:00:30.809469 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:30.809445 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c73aea8-611d-4762-b2d2-6528eeb7dce9-credential-socket\") pod \"e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-gxcgh\" (UID: \"0c73aea8-611d-4762-b2d2-6528eeb7dce9\") " pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-gxcgh" Oct 30 22:00:30.810167 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:30.810143 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c73aea8-611d-4762-b2d2-6528eeb7dce9-istio-data\") pod \"e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-gxcgh\" (UID: \"0c73aea8-611d-4762-b2d2-6528eeb7dce9\") " pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-gxcgh" Oct 30 22:00:30.810296 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:30.810244 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c73aea8-611d-4762-b2d2-6528eeb7dce9-istio-envoy\") pod \"e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-gxcgh\" (UID: \"0c73aea8-611d-4762-b2d2-6528eeb7dce9\") " pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-gxcgh" Oct 30 22:00:30.810296 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:30.810274 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c73aea8-611d-4762-b2d2-6528eeb7dce9-istiod-ca-cert\") pod \"e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-gxcgh\" (UID: \"0c73aea8-611d-4762-b2d2-6528eeb7dce9\") " pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-gxcgh" Oct 30 22:00:30.810493 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:30.810355 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c73aea8-611d-4762-b2d2-6528eeb7dce9-workload-socket\") pod \"e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-gxcgh\" (UID: \"0c73aea8-611d-4762-b2d2-6528eeb7dce9\") " pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-gxcgh" Oct 30 22:00:30.810574 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:30.810248 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c73aea8-611d-4762-b2d2-6528eeb7dce9-workload-certs\") pod \"e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-gxcgh\" (UID: \"0c73aea8-611d-4762-b2d2-6528eeb7dce9\") " pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-gxcgh" Oct 30 22:00:30.811228 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:30.811205 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c73aea8-611d-4762-b2d2-6528eeb7dce9-istiod-ca-cert\") pod \"e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-gxcgh\" (UID: \"0c73aea8-611d-4762-b2d2-6528eeb7dce9\") " pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-gxcgh" Oct 30 22:00:30.811700 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:30.811595 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c73aea8-611d-4762-b2d2-6528eeb7dce9-workload-socket\") pod \"e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-gxcgh\" (UID: \"0c73aea8-611d-4762-b2d2-6528eeb7dce9\") " pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-gxcgh" Oct 30 22:00:30.812316 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:30.812292 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c73aea8-611d-4762-b2d2-6528eeb7dce9-istio-podinfo\") pod \"e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-gxcgh\" (UID: \"0c73aea8-611d-4762-b2d2-6528eeb7dce9\") " pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-gxcgh" Oct 30 22:00:30.812636 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:30.812618 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c73aea8-611d-4762-b2d2-6528eeb7dce9-istio-envoy\") pod \"e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-gxcgh\" (UID: \"0c73aea8-611d-4762-b2d2-6528eeb7dce9\") " pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-gxcgh" Oct 30 22:00:30.822660 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:30.822639 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c73aea8-611d-4762-b2d2-6528eeb7dce9-istio-token\") pod \"e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-gxcgh\" (UID: \"0c73aea8-611d-4762-b2d2-6528eeb7dce9\") " pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-gxcgh" Oct 30 22:00:30.822886 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:30.822864 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5j29\" (UniqueName: \"kubernetes.io/projected/0c73aea8-611d-4762-b2d2-6528eeb7dce9-kube-api-access-d5j29\") pod \"e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-gxcgh\" (UID: \"0c73aea8-611d-4762-b2d2-6528eeb7dce9\") " pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-gxcgh" Oct 30 22:00:31.011764 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:31.011726 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-gxcgh" Oct 30 22:00:31.313741 ip-10-0-137-181 kubenswrapper[6720]: W1030 22:00:31.313705 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c73aea8_611d_4762_b2d2_6528eeb7dce9.slice/crio-4e587fcdee791b417719c4de40503562cdf736d94929cb151606afd2585664ba WatchSource:0}: Error finding container 4e587fcdee791b417719c4de40503562cdf736d94929cb151606afd2585664ba: Status 404 returned error can't find the container with id 4e587fcdee791b417719c4de40503562cdf736d94929cb151606afd2585664ba Oct 30 22:00:31.313908 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:31.313808 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-gxcgh"] Oct 30 22:00:31.317667 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:31.317628 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:00:31.317781 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:31.317701 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:00:31.317781 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:31.317735 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:00:31.737543 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:31.737493 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-gxcgh" event={"ID":"0c73aea8-611d-4762-b2d2-6528eeb7dce9","Type":"ContainerStarted","Data":"152ea05b537bd0851a95d5a79703a6306ef729a7bae644977b7b61ecdd88327c"} Oct 30 22:00:31.737998 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:31.737928 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-gxcgh" event={"ID":"0c73aea8-611d-4762-b2d2-6528eeb7dce9","Type":"ContainerStarted","Data":"4e587fcdee791b417719c4de40503562cdf736d94929cb151606afd2585664ba"} Oct 30 22:00:31.763559 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:31.763526 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:31.769426 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:31.769352 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-gxcgh" podStartSLOduration=1.769333184 podStartE2EDuration="1.769333184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:31.768119695 +0000 UTC m=+1197.939084910" watchObservedRunningTime="2026-10-30 22:00:31.769333184 +0000 UTC m=+1197.940298372" Oct 30 22:00:32.012294 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:32.012217 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-gxcgh" Oct 30 22:00:32.013869 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:32.013842 6720 patch_prober.go:29] interesting pod/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-gxcgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 22:00:32.013992 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:32.013910 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-gxcgh" podUID="0c73aea8-611d-4762-b2d2-6528eeb7dce9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 22:00:32.476411 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:32.475631 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:33.012728 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:33.012692 6720 patch_prober.go:29] interesting pod/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-gxcgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 22:00:33.013129 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:33.012778 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-gxcgh" podUID="0c73aea8-611d-4762-b2d2-6528eeb7dce9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 22:00:33.042926 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:33.042893 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:33.217621 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:33.217589 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:33.403334 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:33.403303 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:33.579465 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:33.577826 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:34.012827 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:34.012794 6720 patch_prober.go:29] interesting pod/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-gxcgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 22:00:34.013204 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:34.012841 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-gxcgh" podUID="0c73aea8-611d-4762-b2d2-6528eeb7dce9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 22:00:34.304334 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:34.303704 6720 scope.go:122] "RemoveContainer" containerID="2a9f5940ba3d075223d6fefcff0e9bf0a1927aa83677401faf19d577ba1c5d29" Oct 30 22:00:34.572469 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:34.571730 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:34.770052 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:34.770015 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:35.009020 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:35.008991 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:35.012376 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:35.012351 6720 patch_prober.go:29] interesting pod/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-gxcgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 22:00:35.012464 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:35.012421 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-gxcgh" podUID="0c73aea8-611d-4762-b2d2-6528eeb7dce9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 22:00:35.646418 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:35.644448 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:36.012113 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:36.012023 6720 patch_prober.go:29] interesting pod/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-gxcgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 22:00:36.012113 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:36.012073 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-gxcgh" podUID="0c73aea8-611d-4762-b2d2-6528eeb7dce9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 22:00:36.570367 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:36.570330 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:37.012547 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:37.012512 6720 patch_prober.go:29] interesting pod/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-gxcgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 22:00:37.012925 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:37.012573 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-gxcgh" podUID="0c73aea8-611d-4762-b2d2-6528eeb7dce9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 22:00:37.437694 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:37.437660 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:38.017606 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:38.017578 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-gxcgh" Oct 30 22:00:38.018057 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:38.018005 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-gxcgh" Oct 30 22:00:38.019319 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:38.019298 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-gxcgh" Oct 30 22:00:38.443442 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:38.443379 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:39.146737 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:39.146698 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:40.077254 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:40.077207 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:40.403647 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:00:40.403615 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 22:00:40.623314 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:40.623274 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:40.777833 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:40.777753 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:41.238135 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:41.238099 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:41.848284 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:41.848247 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:42.541368 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:42.541312 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:43.395782 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:43.395726 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:44.010467 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:44.010431 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:44.820014 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:44.819981 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:45.614129 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:45.614080 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:46.487338 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:46.487304 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:47.348213 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:47.348181 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:47.687818 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:47.687025 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:47.907099 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:47.907062 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:48.278468 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:48.278435 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:48.890442 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:48.890364 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:49.104434 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:49.104371 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:49.387405 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:49.384492 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:49.979350 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:49.979316 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:50.558213 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:50.558178 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:50.784340 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:50.784305 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:51.120910 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:51.120877 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:51.427813 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:51.427714 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:51.614736 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:51.614701 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:52.591010 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:52.590974 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:52.920560 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:52.920532 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269"] Oct 30 22:00:52.920744 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:52.920719 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269" podUID="01205782-af7b-4eda-894d-3850de540135" containerName="istio-proxy" containerID="cri-o://d96ef1c5779215cd09940284bd4b102d5ee5fcdd28f6c870dd4fcdca37681940" gracePeriod=30 Oct 30 22:00:53.643421 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:53.628632 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:54.555803 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:54.555721 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:55.200914 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:55.200876 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:55.454598 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:55.454485 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:55.903674 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:55.903643 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:56.597091 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:56.597057 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:56.922696 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:56.922659 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:57.233552 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:57.233462 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:58.115509 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:58.115472 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:58.194725 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:58.194665 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269" Oct 30 22:00:58.277415 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:58.277333 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:58.375562 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:58.375524 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01205782-af7b-4eda-894d-3850de540135-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01205782-af7b-4eda-894d-3850de540135-istiod-ca-cert\") pod \"01205782-af7b-4eda-894d-3850de540135\" (UID: \"01205782-af7b-4eda-894d-3850de540135\") " Oct 30 22:00:58.375752 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:58.375591 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01205782-af7b-4eda-894d-3850de540135-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01205782-af7b-4eda-894d-3850de540135-workload-socket\") pod \"01205782-af7b-4eda-894d-3850de540135\" (UID: \"01205782-af7b-4eda-894d-3850de540135\") " Oct 30 22:00:58.375752 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:58.375690 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/01205782-af7b-4eda-894d-3850de540135-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01205782-af7b-4eda-894d-3850de540135-istio-podinfo\") pod \"01205782-af7b-4eda-894d-3850de540135\" (UID: \"01205782-af7b-4eda-894d-3850de540135\") " Oct 30 22:00:58.375870 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:58.375841 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01205782-af7b-4eda-894d-3850de540135-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01205782-af7b-4eda-894d-3850de540135-workload-certs\") pod \"01205782-af7b-4eda-894d-3850de540135\" (UID: \"01205782-af7b-4eda-894d-3850de540135\") " Oct 30 22:00:58.375929 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:58.375908 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01205782-af7b-4eda-894d-3850de540135-kube-api-access-6jrtr\" (UniqueName: \"kubernetes.io/projected/01205782-af7b-4eda-894d-3850de540135-kube-api-access-6jrtr\") pod \"01205782-af7b-4eda-894d-3850de540135\" (UID: \"01205782-af7b-4eda-894d-3850de540135\") " Oct 30 22:00:58.375984 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:58.375972 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01205782-af7b-4eda-894d-3850de540135-istio-token\" (UniqueName: \"kubernetes.io/projected/01205782-af7b-4eda-894d-3850de540135-istio-token\") pod \"01205782-af7b-4eda-894d-3850de540135\" (UID: \"01205782-af7b-4eda-894d-3850de540135\") " Oct 30 22:00:58.376037 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:58.376000 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01205782-af7b-4eda-894d-3850de540135-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01205782-af7b-4eda-894d-3850de540135-credential-socket\") pod \"01205782-af7b-4eda-894d-3850de540135\" (UID: \"01205782-af7b-4eda-894d-3850de540135\") " Oct 30 22:00:58.376090 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:58.376041 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01205782-af7b-4eda-894d-3850de540135-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01205782-af7b-4eda-894d-3850de540135-istio-data\") pod \"01205782-af7b-4eda-894d-3850de540135\" (UID: \"01205782-af7b-4eda-894d-3850de540135\") " Oct 30 22:00:58.376090 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:58.376069 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01205782-af7b-4eda-894d-3850de540135-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01205782-af7b-4eda-894d-3850de540135-istio-envoy\") pod \"01205782-af7b-4eda-894d-3850de540135\" (UID: \"01205782-af7b-4eda-894d-3850de540135\") " Oct 30 22:00:58.378018 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:58.375901 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01205782-af7b-4eda-894d-3850de540135-workload-socket" pod "01205782-af7b-4eda-894d-3850de540135" (UID: "01205782-af7b-4eda-894d-3850de540135"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:00:58.378018 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:58.375973 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01205782-af7b-4eda-894d-3850de540135-istiod-ca-cert" pod "01205782-af7b-4eda-894d-3850de540135" (UID: "01205782-af7b-4eda-894d-3850de540135"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:00:58.378018 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:58.376342 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01205782-af7b-4eda-894d-3850de540135-workload-certs" pod "01205782-af7b-4eda-894d-3850de540135" (UID: "01205782-af7b-4eda-894d-3850de540135"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:00:58.378018 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:58.376504 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01205782-af7b-4eda-894d-3850de540135-credential-socket" pod "01205782-af7b-4eda-894d-3850de540135" (UID: "01205782-af7b-4eda-894d-3850de540135"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:00:58.378018 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:58.376715 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01205782-af7b-4eda-894d-3850de540135-istio-data" pod "01205782-af7b-4eda-894d-3850de540135" (UID: "01205782-af7b-4eda-894d-3850de540135"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:00:58.378018 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:58.377903 6720 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01205782-af7b-4eda-894d-3850de540135-workload-certs\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:58.378018 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:58.377947 6720 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01205782-af7b-4eda-894d-3850de540135-credential-socket\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:58.378018 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:58.377965 6720 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01205782-af7b-4eda-894d-3850de540135-istio-data\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:58.378018 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:58.377979 6720 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01205782-af7b-4eda-894d-3850de540135-istiod-ca-cert\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:58.378018 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:58.377992 6720 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01205782-af7b-4eda-894d-3850de540135-workload-socket\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:58.378949 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:58.378924 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01205782-af7b-4eda-894d-3850de540135-istio-token" pod "01205782-af7b-4eda-894d-3850de540135" (UID: "01205782-af7b-4eda-894d-3850de540135"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:58.381730 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:58.381703 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/01205782-af7b-4eda-894d-3850de540135-istio-podinfo" pod "01205782-af7b-4eda-894d-3850de540135" (UID: "01205782-af7b-4eda-894d-3850de540135"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:00:58.381828 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:58.381762 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01205782-af7b-4eda-894d-3850de540135-kube-api-access-6jrtr" pod "01205782-af7b-4eda-894d-3850de540135" (UID: "01205782-af7b-4eda-894d-3850de540135"). InnerVolumeSpecName "kube-api-access-6jrtr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:00:58.381883 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:58.381865 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01205782-af7b-4eda-894d-3850de540135-istio-envoy" pod "01205782-af7b-4eda-894d-3850de540135" (UID: "01205782-af7b-4eda-894d-3850de540135"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:00:58.479366 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:58.479271 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jrtr\" (UniqueName: \"kubernetes.io/projected/01205782-af7b-4eda-894d-3850de540135-kube-api-access-6jrtr\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:58.479366 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:58.479310 6720 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01205782-af7b-4eda-894d-3850de540135-istio-token\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:58.479366 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:58.479324 6720 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01205782-af7b-4eda-894d-3850de540135-istio-envoy\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:58.479366 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:58.479336 6720 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01205782-af7b-4eda-894d-3850de540135-istio-podinfo\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:00:58.587409 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:58.570595 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:59.012814 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:59.012781 6720 generic.go:356] "Generic (PLEG): container finished" podID="01205782-af7b-4eda-894d-3850de540135" containerID="d96ef1c5779215cd09940284bd4b102d5ee5fcdd28f6c870dd4fcdca37681940" exitCode=0 Oct 30 22:00:59.013016 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:59.012826 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269" Oct 30 22:00:59.013016 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:59.012875 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269" event={"ID":"01205782-af7b-4eda-894d-3850de540135","Type":"ContainerDied","Data":"d96ef1c5779215cd09940284bd4b102d5ee5fcdd28f6c870dd4fcdca37681940"} Oct 30 22:00:59.013016 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:59.012932 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269" event={"ID":"01205782-af7b-4eda-894d-3850de540135","Type":"ContainerDied","Data":"312cbe52ade4cdb38ea8d0398a1d742d4495fd2508c18d03d8c67983b4faecc4"} Oct 30 22:00:59.013016 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:59.012963 6720 scope.go:122] "RemoveContainer" containerID="d96ef1c5779215cd09940284bd4b102d5ee5fcdd28f6c870dd4fcdca37681940" Oct 30 22:00:59.025345 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:59.025326 6720 scope.go:122] "RemoveContainer" containerID="d96ef1c5779215cd09940284bd4b102d5ee5fcdd28f6c870dd4fcdca37681940" Oct 30 22:00:59.025786 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:00:59.025753 6720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d96ef1c5779215cd09940284bd4b102d5ee5fcdd28f6c870dd4fcdca37681940\": container with ID starting with d96ef1c5779215cd09940284bd4b102d5ee5fcdd28f6c870dd4fcdca37681940 not found: ID does not exist" containerID="d96ef1c5779215cd09940284bd4b102d5ee5fcdd28f6c870dd4fcdca37681940" Oct 30 22:00:59.025906 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:59.025786 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d96ef1c5779215cd09940284bd4b102d5ee5fcdd28f6c870dd4fcdca37681940"} err="failed to get container status \"d96ef1c5779215cd09940284bd4b102d5ee5fcdd28f6c870dd4fcdca37681940\": rpc error: code = NotFound desc = could not find container \"d96ef1c5779215cd09940284bd4b102d5ee5fcdd28f6c870dd4fcdca37681940\": container with ID starting with d96ef1c5779215cd09940284bd4b102d5ee5fcdd28f6c870dd4fcdca37681940 not found: ID does not exist" Oct 30 22:00:59.045882 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:59.045848 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269"] Oct 30 22:00:59.048531 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:59.048509 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-2n269"] Oct 30 22:00:59.289353 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:59.289266 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:59.494288 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:59.494197 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:00:59.669302 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:00:59.669258 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:00.011182 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:00.011108 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01205782-af7b-4eda-894d-3850de540135" path="/var/lib/kubelet/pods/01205782-af7b-4eda-894d-3850de540135/volumes" Oct 30 22:01:00.309331 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:00.309190 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:01.312960 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:01.312685 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:01.550629 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:01.550591 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:02.129696 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:02.129655 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:02.588144 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:02.588056 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:02.788624 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:02.788591 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:03.417488 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:03.417452 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:04.170042 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:04.170011 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:05.035229 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:05.035168 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:05.483431 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:05.483401 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:05.645238 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:05.645205 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:05.933343 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:05.933308 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:06.786339 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:06.786304 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:07.011410 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:06.996297 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:07.185245 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:07.185208 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:07.649089 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:07.649042 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:07.999240 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:07.999144 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:08.950092 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:08.950063 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:09.773722 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:09.771425 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:10.407145 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:01:10.407103 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 22:01:10.657676 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:10.657577 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:11.436321 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:11.436290 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:12.147256 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:12.147220 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:12.619048 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:12.619020 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-19cab1-5855fdfb65-zxd9n"] Oct 30 22:01:12.619462 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:12.619230 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-19cab1-5855fdfb65-zxd9n" podUID="64d0f5bd-9b36-4ec4-b3c3-d47976e760e9" containerName="maas-api" containerID="cri-o://342e9e768371b05c5895868ce9e338201735f509054a29ba9063deafad5dbb68" gracePeriod=30 Oct 30 22:01:12.703173 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:12.703142 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:12.881480 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:12.881301 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:13.115058 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:13.107265 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:13.117499 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:13.117460 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-19cab1-5855fdfb65-zxd9n" Oct 30 22:01:13.156568 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:13.156483 6720 generic.go:356] "Generic (PLEG): container finished" podID="64d0f5bd-9b36-4ec4-b3c3-d47976e760e9" containerID="342e9e768371b05c5895868ce9e338201735f509054a29ba9063deafad5dbb68" exitCode=0 Oct 30 22:01:13.156568 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:13.156553 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-19cab1-5855fdfb65-zxd9n" Oct 30 22:01:13.156898 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:13.156694 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-19cab1-5855fdfb65-zxd9n" event={"ID":"64d0f5bd-9b36-4ec4-b3c3-d47976e760e9","Type":"ContainerDied","Data":"342e9e768371b05c5895868ce9e338201735f509054a29ba9063deafad5dbb68"} Oct 30 22:01:13.156898 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:13.156745 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-19cab1-5855fdfb65-zxd9n" event={"ID":"64d0f5bd-9b36-4ec4-b3c3-d47976e760e9","Type":"ContainerDied","Data":"f2bd04dbdc10e443923f6dff056bcc66d80c17d4de163a15577ae602a86102f9"} Oct 30 22:01:13.156898 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:13.156775 6720 scope.go:122] "RemoveContainer" containerID="342e9e768371b05c5895868ce9e338201735f509054a29ba9063deafad5dbb68" Oct 30 22:01:13.184143 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:13.184116 6720 scope.go:122] "RemoveContainer" containerID="342e9e768371b05c5895868ce9e338201735f509054a29ba9063deafad5dbb68" Oct 30 22:01:13.184709 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:01:13.184677 6720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"342e9e768371b05c5895868ce9e338201735f509054a29ba9063deafad5dbb68\": container with ID starting with 342e9e768371b05c5895868ce9e338201735f509054a29ba9063deafad5dbb68 not found: ID does not exist" containerID="342e9e768371b05c5895868ce9e338201735f509054a29ba9063deafad5dbb68" Oct 30 22:01:13.184871 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:13.184734 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"342e9e768371b05c5895868ce9e338201735f509054a29ba9063deafad5dbb68"} err="failed to get container status \"342e9e768371b05c5895868ce9e338201735f509054a29ba9063deafad5dbb68\": rpc error: code = NotFound desc = could not find container \"342e9e768371b05c5895868ce9e338201735f509054a29ba9063deafad5dbb68\": container with ID starting with 342e9e768371b05c5895868ce9e338201735f509054a29ba9063deafad5dbb68 not found: ID does not exist" Oct 30 22:01:13.220098 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:13.220069 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64d0f5bd-9b36-4ec4-b3c3-d47976e760e9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64d0f5bd-9b36-4ec4-b3c3-d47976e760e9-openshift-service-ca\") pod \"64d0f5bd-9b36-4ec4-b3c3-d47976e760e9\" (UID: \"64d0f5bd-9b36-4ec4-b3c3-d47976e760e9\") " Oct 30 22:01:13.220270 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:13.220155 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64d0f5bd-9b36-4ec4-b3c3-d47976e760e9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/64d0f5bd-9b36-4ec4-b3c3-d47976e760e9-metrics-certs\") pod \"64d0f5bd-9b36-4ec4-b3c3-d47976e760e9\" (UID: \"64d0f5bd-9b36-4ec4-b3c3-d47976e760e9\") " Oct 30 22:01:13.220429 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:13.220410 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64d0f5bd-9b36-4ec4-b3c3-d47976e760e9-kube-api-access-64sqf\" (UniqueName: \"kubernetes.io/projected/64d0f5bd-9b36-4ec4-b3c3-d47976e760e9-kube-api-access-64sqf\") pod \"64d0f5bd-9b36-4ec4-b3c3-d47976e760e9\" (UID: \"64d0f5bd-9b36-4ec4-b3c3-d47976e760e9\") " Oct 30 22:01:13.220495 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:13.220482 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64d0f5bd-9b36-4ec4-b3c3-d47976e760e9-openshift-service-ca" pod "64d0f5bd-9b36-4ec4-b3c3-d47976e760e9" (UID: "64d0f5bd-9b36-4ec4-b3c3-d47976e760e9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:01:13.220553 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:13.220529 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64d0f5bd-9b36-4ec4-b3c3-d47976e760e9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64d0f5bd-9b36-4ec4-b3c3-d47976e760e9-maas-api-tls\") pod \"64d0f5bd-9b36-4ec4-b3c3-d47976e760e9\" (UID: \"64d0f5bd-9b36-4ec4-b3c3-d47976e760e9\") " Oct 30 22:01:13.221038 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:13.220967 6720 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64d0f5bd-9b36-4ec4-b3c3-d47976e760e9-openshift-service-ca\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:13.222565 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:13.222533 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64d0f5bd-9b36-4ec4-b3c3-d47976e760e9-metrics-certs" pod "64d0f5bd-9b36-4ec4-b3c3-d47976e760e9" (UID: "64d0f5bd-9b36-4ec4-b3c3-d47976e760e9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:01:13.222659 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:13.222565 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64d0f5bd-9b36-4ec4-b3c3-d47976e760e9-kube-api-access-64sqf" pod "64d0f5bd-9b36-4ec4-b3c3-d47976e760e9" (UID: "64d0f5bd-9b36-4ec4-b3c3-d47976e760e9"). InnerVolumeSpecName "kube-api-access-64sqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:01:13.222794 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:13.222770 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64d0f5bd-9b36-4ec4-b3c3-d47976e760e9-maas-api-tls" pod "64d0f5bd-9b36-4ec4-b3c3-d47976e760e9" (UID: "64d0f5bd-9b36-4ec4-b3c3-d47976e760e9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:01:13.322764 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:13.322728 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64sqf\" (UniqueName: \"kubernetes.io/projected/64d0f5bd-9b36-4ec4-b3c3-d47976e760e9-kube-api-access-64sqf\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:13.323024 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:13.322793 6720 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64d0f5bd-9b36-4ec4-b3c3-d47976e760e9-maas-api-tls\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:13.323024 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:13.322812 6720 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64d0f5bd-9b36-4ec4-b3c3-d47976e760e9-metrics-certs\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:13.324551 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:13.324525 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:13.483764 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:13.483731 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-19cab1-5855fdfb65-zxd9n"] Oct 30 22:01:13.492266 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:13.492238 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-19cab1-5855fdfb65-zxd9n"] Oct 30 22:01:13.497242 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:13.497220 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:13.719020 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:13.718941 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:13.959116 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:13.959076 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64d0f5bd-9b36-4ec4-b3c3-d47976e760e9" path="/var/lib/kubelet/pods/64d0f5bd-9b36-4ec4-b3c3-d47976e760e9/volumes" Oct 30 22:01:14.252350 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:14.252313 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:15.205103 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:15.205073 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:16.185212 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:16.176769 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:16.347270 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:16.347239 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:17.088672 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:17.086333 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:18.042974 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:18.042940 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:18.846997 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:18.846962 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:19.252715 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:19.252676 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c614f9-69b7c4fb64-kgzkp"] Oct 30 22:01:19.253320 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:19.253292 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c614f9-69b7c4fb64-kgzkp" podUID="5f8bf96d-b5df-4775-9d77-a725a0ca6a2b" containerName="maas-api" containerID="cri-o://83c1cdc4f3de904a53626c55b6e1281e005d12be3f7001cf5c44da27d451bf09" gracePeriod=30 Oct 30 22:01:19.347866 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:19.347831 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:19.508474 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:19.507783 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:19.659410 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:19.657577 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c614f9-69b7c4fb64-kgzkp" Oct 30 22:01:19.720445 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:19.709272 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:19.834011 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:19.833921 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f8bf96d-b5df-4775-9d77-a725a0ca6a2b-kube-api-access-ggqm2\" (UniqueName: \"kubernetes.io/projected/5f8bf96d-b5df-4775-9d77-a725a0ca6a2b-kube-api-access-ggqm2\") pod \"5f8bf96d-b5df-4775-9d77-a725a0ca6a2b\" (UID: \"5f8bf96d-b5df-4775-9d77-a725a0ca6a2b\") " Oct 30 22:01:19.834011 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:19.833973 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f8bf96d-b5df-4775-9d77-a725a0ca6a2b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f8bf96d-b5df-4775-9d77-a725a0ca6a2b-maas-api-tls\") pod \"5f8bf96d-b5df-4775-9d77-a725a0ca6a2b\" (UID: \"5f8bf96d-b5df-4775-9d77-a725a0ca6a2b\") " Oct 30 22:01:19.834611 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:19.834188 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f8bf96d-b5df-4775-9d77-a725a0ca6a2b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f8bf96d-b5df-4775-9d77-a725a0ca6a2b-openshift-service-ca\") pod \"5f8bf96d-b5df-4775-9d77-a725a0ca6a2b\" (UID: \"5f8bf96d-b5df-4775-9d77-a725a0ca6a2b\") " Oct 30 22:01:19.834611 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:19.834276 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f8bf96d-b5df-4775-9d77-a725a0ca6a2b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f8bf96d-b5df-4775-9d77-a725a0ca6a2b-metrics-certs\") pod \"5f8bf96d-b5df-4775-9d77-a725a0ca6a2b\" (UID: \"5f8bf96d-b5df-4775-9d77-a725a0ca6a2b\") " Oct 30 22:01:19.835007 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:19.834978 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f8bf96d-b5df-4775-9d77-a725a0ca6a2b-openshift-service-ca" pod "5f8bf96d-b5df-4775-9d77-a725a0ca6a2b" (UID: "5f8bf96d-b5df-4775-9d77-a725a0ca6a2b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:01:19.836871 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:19.836830 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f8bf96d-b5df-4775-9d77-a725a0ca6a2b-maas-api-tls" pod "5f8bf96d-b5df-4775-9d77-a725a0ca6a2b" (UID: "5f8bf96d-b5df-4775-9d77-a725a0ca6a2b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:01:19.837805 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:19.837782 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f8bf96d-b5df-4775-9d77-a725a0ca6a2b-metrics-certs" pod "5f8bf96d-b5df-4775-9d77-a725a0ca6a2b" (UID: "5f8bf96d-b5df-4775-9d77-a725a0ca6a2b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:01:19.839491 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:19.839464 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f8bf96d-b5df-4775-9d77-a725a0ca6a2b-kube-api-access-ggqm2" pod "5f8bf96d-b5df-4775-9d77-a725a0ca6a2b" (UID: "5f8bf96d-b5df-4775-9d77-a725a0ca6a2b"). InnerVolumeSpecName "kube-api-access-ggqm2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:01:19.881763 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:19.881722 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:19.936545 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:19.936506 6720 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f8bf96d-b5df-4775-9d77-a725a0ca6a2b-openshift-service-ca\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:19.936545 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:19.936548 6720 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f8bf96d-b5df-4775-9d77-a725a0ca6a2b-metrics-certs\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:19.936775 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:19.936569 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggqm2\" (UniqueName: \"kubernetes.io/projected/5f8bf96d-b5df-4775-9d77-a725a0ca6a2b-kube-api-access-ggqm2\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:19.936775 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:19.936589 6720 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f8bf96d-b5df-4775-9d77-a725a0ca6a2b-maas-api-tls\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:20.041919 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:20.041884 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:20.203867 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:20.203835 6720 generic.go:356] "Generic (PLEG): container finished" podID="5f8bf96d-b5df-4775-9d77-a725a0ca6a2b" containerID="83c1cdc4f3de904a53626c55b6e1281e005d12be3f7001cf5c44da27d451bf09" exitCode=0 Oct 30 22:01:20.204038 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:20.203903 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c614f9-69b7c4fb64-kgzkp" event={"ID":"5f8bf96d-b5df-4775-9d77-a725a0ca6a2b","Type":"ContainerDied","Data":"83c1cdc4f3de904a53626c55b6e1281e005d12be3f7001cf5c44da27d451bf09"} Oct 30 22:01:20.204038 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:20.203961 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c614f9-69b7c4fb64-kgzkp" Oct 30 22:01:20.204038 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:20.203968 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c614f9-69b7c4fb64-kgzkp" event={"ID":"5f8bf96d-b5df-4775-9d77-a725a0ca6a2b","Type":"ContainerDied","Data":"bc71e985ac5dbe392b547a79cf836d13c9f5e9b2fffefbc0c4a4fbb0b26b2b09"} Oct 30 22:01:20.204038 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:20.203998 6720 scope.go:122] "RemoveContainer" containerID="83c1cdc4f3de904a53626c55b6e1281e005d12be3f7001cf5c44da27d451bf09" Oct 30 22:01:20.215011 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:20.214990 6720 scope.go:122] "RemoveContainer" containerID="83c1cdc4f3de904a53626c55b6e1281e005d12be3f7001cf5c44da27d451bf09" Oct 30 22:01:20.215254 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:01:20.215233 6720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83c1cdc4f3de904a53626c55b6e1281e005d12be3f7001cf5c44da27d451bf09\": container with ID starting with 83c1cdc4f3de904a53626c55b6e1281e005d12be3f7001cf5c44da27d451bf09 not found: ID does not exist" containerID="83c1cdc4f3de904a53626c55b6e1281e005d12be3f7001cf5c44da27d451bf09" Oct 30 22:01:20.215347 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:20.215266 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83c1cdc4f3de904a53626c55b6e1281e005d12be3f7001cf5c44da27d451bf09"} err="failed to get container status \"83c1cdc4f3de904a53626c55b6e1281e005d12be3f7001cf5c44da27d451bf09\": rpc error: code = NotFound desc = could not find container \"83c1cdc4f3de904a53626c55b6e1281e005d12be3f7001cf5c44da27d451bf09\": container with ID starting with 83c1cdc4f3de904a53626c55b6e1281e005d12be3f7001cf5c44da27d451bf09 not found: ID does not exist" Oct 30 22:01:20.225434 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:20.225367 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c614f9-69b7c4fb64-kgzkp"] Oct 30 22:01:20.233484 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:20.233456 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c614f9-69b7c4fb64-kgzkp"] Oct 30 22:01:20.412659 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:20.412623 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:20.559578 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:20.559469 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:20.750249 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:20.750206 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:21.098286 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:21.098251 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:21.659407 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:21.658916 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:21.958536 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:21.958460 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f8bf96d-b5df-4775-9d77-a725a0ca6a2b" path="/var/lib/kubelet/pods/5f8bf96d-b5df-4775-9d77-a725a0ca6a2b/volumes" Oct 30 22:01:22.317871 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:22.317780 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:22.653433 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:22.653377 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:23.189411 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:23.178333 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:23.406186 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:23.406139 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:24.035970 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:24.035936 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:25.334180 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:25.334145 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:26.372299 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:26.372256 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:26.620943 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:26.620902 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:26.916126 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:26.916085 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:27.227445 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:27.227361 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:27.436657 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:27.436620 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:27.767267 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:27.767238 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:27.986703 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:27.986670 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:28.826119 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:28.826088 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:29.545182 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:29.545146 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:29.765677 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:29.765650 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:30.067988 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:30.067958 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:30.773977 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:30.773942 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:30.923925 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:30.923894 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:31.638648 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:31.638612 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:31.983876 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:31.983799 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:32.334454 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:32.332523 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3654d5-fc45498c9-9tgck"] Oct 30 22:01:32.334454 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:32.332816 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3654d5-fc45498c9-9tgck" podUID="b54e2d16-31f2-4c81-8dab-12e86f00f79f" containerName="payload-pre-processing" containerID="cri-o://d549fce84e231092ace93948567579bab651132b43598c13ee5b62ed48908f90" gracePeriod=30 Oct 30 22:01:32.341680 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:32.341618 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:32.533757 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:32.533722 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:32.713411 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:32.712425 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:32.886734 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:32.886709 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:32.920162 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:32.920131 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3654d5-fc45498c9-9tgck" Oct 30 22:01:33.028481 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:33.028363 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:33.065747 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:33.065717 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b54e2d16-31f2-4c81-8dab-12e86f00f79f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b54e2d16-31f2-4c81-8dab-12e86f00f79f-plugins-config-volume\") pod \"b54e2d16-31f2-4c81-8dab-12e86f00f79f\" (UID: \"b54e2d16-31f2-4c81-8dab-12e86f00f79f\") " Oct 30 22:01:33.065993 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:33.065840 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b54e2d16-31f2-4c81-8dab-12e86f00f79f-kube-api-access-tmfz8\" (UniqueName: \"kubernetes.io/projected/b54e2d16-31f2-4c81-8dab-12e86f00f79f-kube-api-access-tmfz8\") pod \"b54e2d16-31f2-4c81-8dab-12e86f00f79f\" (UID: \"b54e2d16-31f2-4c81-8dab-12e86f00f79f\") " Oct 30 22:01:33.066104 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:33.066080 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b54e2d16-31f2-4c81-8dab-12e86f00f79f-plugins-config-volume" pod "b54e2d16-31f2-4c81-8dab-12e86f00f79f" (UID: "b54e2d16-31f2-4c81-8dab-12e86f00f79f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:01:33.066491 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:33.066467 6720 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b54e2d16-31f2-4c81-8dab-12e86f00f79f-plugins-config-volume\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:33.068464 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:33.068441 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b54e2d16-31f2-4c81-8dab-12e86f00f79f-kube-api-access-tmfz8" pod "b54e2d16-31f2-4c81-8dab-12e86f00f79f" (UID: "b54e2d16-31f2-4c81-8dab-12e86f00f79f"). InnerVolumeSpecName "kube-api-access-tmfz8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:01:33.168039 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:33.168005 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmfz8\" (UniqueName: \"kubernetes.io/projected/b54e2d16-31f2-4c81-8dab-12e86f00f79f-kube-api-access-tmfz8\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:01:33.357086 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:33.357050 6720 generic.go:356] "Generic (PLEG): container finished" podID="b54e2d16-31f2-4c81-8dab-12e86f00f79f" containerID="d549fce84e231092ace93948567579bab651132b43598c13ee5b62ed48908f90" exitCode=0 Oct 30 22:01:33.357276 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:33.357119 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3654d5-fc45498c9-9tgck" Oct 30 22:01:33.357276 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:33.357132 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3654d5-fc45498c9-9tgck" event={"ID":"b54e2d16-31f2-4c81-8dab-12e86f00f79f","Type":"ContainerDied","Data":"d549fce84e231092ace93948567579bab651132b43598c13ee5b62ed48908f90"} Oct 30 22:01:33.357276 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:33.357202 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3654d5-fc45498c9-9tgck" event={"ID":"b54e2d16-31f2-4c81-8dab-12e86f00f79f","Type":"ContainerDied","Data":"4f0c183474a9062da6b1bb8bfe9f649b216d58a5236bbc41a75f347a161bf262"} Oct 30 22:01:33.357276 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:33.357231 6720 scope.go:122] "RemoveContainer" containerID="d549fce84e231092ace93948567579bab651132b43598c13ee5b62ed48908f90" Oct 30 22:01:33.377808 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:33.377691 6720 scope.go:122] "RemoveContainer" containerID="d549fce84e231092ace93948567579bab651132b43598c13ee5b62ed48908f90" Oct 30 22:01:33.378156 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:01:33.378136 6720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d549fce84e231092ace93948567579bab651132b43598c13ee5b62ed48908f90\": container with ID starting with d549fce84e231092ace93948567579bab651132b43598c13ee5b62ed48908f90 not found: ID does not exist" containerID="d549fce84e231092ace93948567579bab651132b43598c13ee5b62ed48908f90" Oct 30 22:01:33.378225 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:33.378165 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d549fce84e231092ace93948567579bab651132b43598c13ee5b62ed48908f90"} err="failed to get container status \"d549fce84e231092ace93948567579bab651132b43598c13ee5b62ed48908f90\": rpc error: code = NotFound desc = could not find container \"d549fce84e231092ace93948567579bab651132b43598c13ee5b62ed48908f90\": container with ID starting with d549fce84e231092ace93948567579bab651132b43598c13ee5b62ed48908f90 not found: ID does not exist" Oct 30 22:01:33.384200 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:33.384173 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3654d5-fc45498c9-9tgck"] Oct 30 22:01:33.391263 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:33.391241 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3654d5-fc45498c9-9tgck"] Oct 30 22:01:33.504997 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:33.504960 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:33.933423 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:33.930465 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:33.958793 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:33.958756 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b54e2d16-31f2-4c81-8dab-12e86f00f79f" path="/var/lib/kubelet/pods/b54e2d16-31f2-4c81-8dab-12e86f00f79f/volumes" Oct 30 22:01:34.555798 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:34.555741 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:34.719410 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:34.708721 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:34.858091 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:34.858057 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:35.699952 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:35.699839 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:36.476518 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:36.476432 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:36.857997 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:36.857957 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:37.085610 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:37.085578 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:37.854292 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:37.854259 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:38.245557 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:38.245486 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:38.454411 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:38.445799 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:38.631157 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:38.631124 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:38.937861 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:38.937779 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:39.706041 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:39.706005 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:40.411083 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:01:40.411037 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 22:01:40.549688 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:40.549647 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:41.241160 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:41.241125 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:41.852032 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:41.851991 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:42.535100 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:42.535062 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:43.195312 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:43.195273 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:43.412682 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:43.410814 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:43.614957 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:43.614915 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:43.810131 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:43.810096 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:44.126602 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:44.126569 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:44.512989 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:44.512858 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:45.283826 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:45.283795 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:45.465969 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:45.465931 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:46.300134 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:46.300097 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:47.176818 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:47.176781 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:48.072113 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:48.072082 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:48.286408 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:48.278669 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:48.457096 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:48.457056 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:48.644194 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:48.644167 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:49.171442 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:49.171406 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:49.401433 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:49.397761 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:50.335222 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:50.335176 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:51.333197 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:51.333162 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:51.803095 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:51.803065 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:52.260477 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:52.260446 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:52.918718 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:52.918682 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:53.142096 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:53.142059 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:53.423306 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:53.423273 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:53.636399 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:53.636345 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:54.642703 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:54.642669 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:55.451942 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:55.451903 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:56.361960 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:56.361930 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:56.979052 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:56.978990 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:57.187021 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:57.186985 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:57.396056 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:57.396027 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:57.659725 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:57.659581 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:57.887623 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:57.887563 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:58.516007 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:58.515958 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:58.734226 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:58.734184 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:58.992124 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:58.992091 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:59.221270 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:59.221236 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:01:59.851989 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:01:59.851950 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:00.034629 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:00.034597 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:00.645354 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:00.645320 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:01.458534 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:01.458502 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:01.816438 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:01.816309 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:02.159056 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:02.158862 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-89jzw"] Oct 30 22:02:02.159568 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:02.159548 6720 cpu_manager.go:414] "containerMap: removing container" podUID="5f8bf96d-b5df-4775-9d77-a725a0ca6a2b" containerName="maas-api" Oct 30 22:02:02.159625 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:02.159570 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f8bf96d-b5df-4775-9d77-a725a0ca6a2b" containerName="maas-api" Oct 30 22:02:02.159625 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:02.159587 6720 cpu_manager.go:414] "containerMap: removing container" podUID="b54e2d16-31f2-4c81-8dab-12e86f00f79f" containerName="payload-pre-processing" Oct 30 22:02:02.159625 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:02.159592 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b54e2d16-31f2-4c81-8dab-12e86f00f79f" containerName="payload-pre-processing" Oct 30 22:02:02.159733 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:02.159630 6720 cpu_manager.go:414] "containerMap: removing container" podUID="e2093daf-0e20-462a-be3f-c65e6a136333" containerName="cleanup" Oct 30 22:02:02.159733 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:02.159637 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2093daf-0e20-462a-be3f-c65e6a136333" containerName="cleanup" Oct 30 22:02:02.159733 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:02.159651 6720 cpu_manager.go:414] "containerMap: removing container" podUID="64d0f5bd-9b36-4ec4-b3c3-d47976e760e9" containerName="maas-api" Oct 30 22:02:02.159733 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:02.159656 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64d0f5bd-9b36-4ec4-b3c3-d47976e760e9" containerName="maas-api" Oct 30 22:02:02.159733 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:02.159663 6720 cpu_manager.go:414] "containerMap: removing container" podUID="01205782-af7b-4eda-894d-3850de540135" containerName="istio-proxy" Oct 30 22:02:02.159733 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:02.159667 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01205782-af7b-4eda-894d-3850de540135" containerName="istio-proxy" Oct 30 22:02:02.159733 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:02.159681 6720 cpu_manager.go:414] "containerMap: removing container" podUID="e04efe02-4bd6-48bf-b953-0ee356f2bc88" containerName="cleanup" Oct 30 22:02:02.159733 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:02.159685 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e04efe02-4bd6-48bf-b953-0ee356f2bc88" containerName="cleanup" Oct 30 22:02:02.160051 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:02.159784 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2093daf-0e20-462a-be3f-c65e6a136333" containerName="cleanup" Oct 30 22:02:02.160051 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:02.159795 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="e04efe02-4bd6-48bf-b953-0ee356f2bc88" containerName="cleanup" Oct 30 22:02:02.160051 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:02.159802 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="64d0f5bd-9b36-4ec4-b3c3-d47976e760e9" containerName="maas-api" Oct 30 22:02:02.160051 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:02.159808 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="01205782-af7b-4eda-894d-3850de540135" containerName="istio-proxy" Oct 30 22:02:02.160051 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:02.159815 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f8bf96d-b5df-4775-9d77-a725a0ca6a2b" containerName="maas-api" Oct 30 22:02:02.160051 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:02.159820 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="b54e2d16-31f2-4c81-8dab-12e86f00f79f" containerName="payload-pre-processing" Oct 30 22:02:02.163608 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:02.163580 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-89jzw" Oct 30 22:02:02.168427 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:02.168360 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-71f39e-openshift-default-dockercfg-bzsk6\"" Oct 30 22:02:02.183582 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:02.183560 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-89jzw"] Oct 30 22:02:02.207866 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:02.207833 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/865819b9-2d02-41ee-9d9a-2f6cabff0970-workload-socket\") pod \"e2e-praxis-a-71f39e-openshift-default-64c95f9f85-89jzw\" (UID: \"865819b9-2d02-41ee-9d9a-2f6cabff0970\") " pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-89jzw" Oct 30 22:02:02.208026 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:02.207886 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/865819b9-2d02-41ee-9d9a-2f6cabff0970-istio-envoy\") pod \"e2e-praxis-a-71f39e-openshift-default-64c95f9f85-89jzw\" (UID: \"865819b9-2d02-41ee-9d9a-2f6cabff0970\") " pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-89jzw" Oct 30 22:02:02.208079 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:02.208040 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/865819b9-2d02-41ee-9d9a-2f6cabff0970-istio-data\") pod \"e2e-praxis-a-71f39e-openshift-default-64c95f9f85-89jzw\" (UID: \"865819b9-2d02-41ee-9d9a-2f6cabff0970\") " pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-89jzw" Oct 30 22:02:02.208123 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:02.208082 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlc8f\" (UniqueName: \"kubernetes.io/projected/865819b9-2d02-41ee-9d9a-2f6cabff0970-kube-api-access-vlc8f\") pod \"e2e-praxis-a-71f39e-openshift-default-64c95f9f85-89jzw\" (UID: \"865819b9-2d02-41ee-9d9a-2f6cabff0970\") " pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-89jzw" Oct 30 22:02:02.208197 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:02.208177 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/865819b9-2d02-41ee-9d9a-2f6cabff0970-istio-podinfo\") pod \"e2e-praxis-a-71f39e-openshift-default-64c95f9f85-89jzw\" (UID: \"865819b9-2d02-41ee-9d9a-2f6cabff0970\") " pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-89jzw" Oct 30 22:02:02.208261 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:02.208246 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/865819b9-2d02-41ee-9d9a-2f6cabff0970-istiod-ca-cert\") pod \"e2e-praxis-a-71f39e-openshift-default-64c95f9f85-89jzw\" (UID: \"865819b9-2d02-41ee-9d9a-2f6cabff0970\") " pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-89jzw" Oct 30 22:02:02.208445 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:02.208424 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/865819b9-2d02-41ee-9d9a-2f6cabff0970-istio-token\") pod \"e2e-praxis-a-71f39e-openshift-default-64c95f9f85-89jzw\" (UID: \"865819b9-2d02-41ee-9d9a-2f6cabff0970\") " pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-89jzw" Oct 30 22:02:02.208532 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:02.208517 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/865819b9-2d02-41ee-9d9a-2f6cabff0970-workload-certs\") pod \"e2e-praxis-a-71f39e-openshift-default-64c95f9f85-89jzw\" (UID: \"865819b9-2d02-41ee-9d9a-2f6cabff0970\") " pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-89jzw" Oct 30 22:02:02.208681 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:02.208662 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/865819b9-2d02-41ee-9d9a-2f6cabff0970-credential-socket\") pod \"e2e-praxis-a-71f39e-openshift-default-64c95f9f85-89jzw\" (UID: \"865819b9-2d02-41ee-9d9a-2f6cabff0970\") " pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-89jzw" Oct 30 22:02:02.309868 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:02.309822 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/865819b9-2d02-41ee-9d9a-2f6cabff0970-credential-socket\") pod \"e2e-praxis-a-71f39e-openshift-default-64c95f9f85-89jzw\" (UID: \"865819b9-2d02-41ee-9d9a-2f6cabff0970\") " pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-89jzw" Oct 30 22:02:02.310058 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:02.309895 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/865819b9-2d02-41ee-9d9a-2f6cabff0970-workload-socket\") pod \"e2e-praxis-a-71f39e-openshift-default-64c95f9f85-89jzw\" (UID: \"865819b9-2d02-41ee-9d9a-2f6cabff0970\") " pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-89jzw" Oct 30 22:02:02.310058 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:02.309922 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/865819b9-2d02-41ee-9d9a-2f6cabff0970-istio-envoy\") pod \"e2e-praxis-a-71f39e-openshift-default-64c95f9f85-89jzw\" (UID: \"865819b9-2d02-41ee-9d9a-2f6cabff0970\") " pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-89jzw" Oct 30 22:02:02.310058 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:02.309950 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/865819b9-2d02-41ee-9d9a-2f6cabff0970-istio-data\") pod \"e2e-praxis-a-71f39e-openshift-default-64c95f9f85-89jzw\" (UID: \"865819b9-2d02-41ee-9d9a-2f6cabff0970\") " pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-89jzw" Oct 30 22:02:02.310058 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:02.309967 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlc8f\" (UniqueName: \"kubernetes.io/projected/865819b9-2d02-41ee-9d9a-2f6cabff0970-kube-api-access-vlc8f\") pod \"e2e-praxis-a-71f39e-openshift-default-64c95f9f85-89jzw\" (UID: \"865819b9-2d02-41ee-9d9a-2f6cabff0970\") " pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-89jzw" Oct 30 22:02:02.310058 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:02.310002 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/865819b9-2d02-41ee-9d9a-2f6cabff0970-istio-podinfo\") pod \"e2e-praxis-a-71f39e-openshift-default-64c95f9f85-89jzw\" (UID: \"865819b9-2d02-41ee-9d9a-2f6cabff0970\") " pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-89jzw" Oct 30 22:02:02.310058 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:02.310037 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/865819b9-2d02-41ee-9d9a-2f6cabff0970-istiod-ca-cert\") pod \"e2e-praxis-a-71f39e-openshift-default-64c95f9f85-89jzw\" (UID: \"865819b9-2d02-41ee-9d9a-2f6cabff0970\") " pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-89jzw" Oct 30 22:02:02.310283 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:02.310114 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/865819b9-2d02-41ee-9d9a-2f6cabff0970-istio-token\") pod \"e2e-praxis-a-71f39e-openshift-default-64c95f9f85-89jzw\" (UID: \"865819b9-2d02-41ee-9d9a-2f6cabff0970\") " pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-89jzw" Oct 30 22:02:02.310283 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:02.310157 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/865819b9-2d02-41ee-9d9a-2f6cabff0970-workload-certs\") pod \"e2e-praxis-a-71f39e-openshift-default-64c95f9f85-89jzw\" (UID: \"865819b9-2d02-41ee-9d9a-2f6cabff0970\") " pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-89jzw" Oct 30 22:02:02.310798 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:02.310571 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/865819b9-2d02-41ee-9d9a-2f6cabff0970-workload-certs\") pod \"e2e-praxis-a-71f39e-openshift-default-64c95f9f85-89jzw\" (UID: \"865819b9-2d02-41ee-9d9a-2f6cabff0970\") " pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-89jzw" Oct 30 22:02:02.310798 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:02.310700 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/865819b9-2d02-41ee-9d9a-2f6cabff0970-istio-data\") pod \"e2e-praxis-a-71f39e-openshift-default-64c95f9f85-89jzw\" (UID: \"865819b9-2d02-41ee-9d9a-2f6cabff0970\") " pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-89jzw" Oct 30 22:02:02.310798 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:02.310785 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/865819b9-2d02-41ee-9d9a-2f6cabff0970-workload-socket\") pod \"e2e-praxis-a-71f39e-openshift-default-64c95f9f85-89jzw\" (UID: \"865819b9-2d02-41ee-9d9a-2f6cabff0970\") " pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-89jzw" Oct 30 22:02:02.311071 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:02.311049 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/865819b9-2d02-41ee-9d9a-2f6cabff0970-credential-socket\") pod \"e2e-praxis-a-71f39e-openshift-default-64c95f9f85-89jzw\" (UID: \"865819b9-2d02-41ee-9d9a-2f6cabff0970\") " pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-89jzw" Oct 30 22:02:02.311506 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:02.311484 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/865819b9-2d02-41ee-9d9a-2f6cabff0970-istiod-ca-cert\") pod \"e2e-praxis-a-71f39e-openshift-default-64c95f9f85-89jzw\" (UID: \"865819b9-2d02-41ee-9d9a-2f6cabff0970\") " pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-89jzw" Oct 30 22:02:02.313450 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:02.313423 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/865819b9-2d02-41ee-9d9a-2f6cabff0970-istio-podinfo\") pod \"e2e-praxis-a-71f39e-openshift-default-64c95f9f85-89jzw\" (UID: \"865819b9-2d02-41ee-9d9a-2f6cabff0970\") " pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-89jzw" Oct 30 22:02:02.313582 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:02.313473 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/865819b9-2d02-41ee-9d9a-2f6cabff0970-istio-envoy\") pod \"e2e-praxis-a-71f39e-openshift-default-64c95f9f85-89jzw\" (UID: \"865819b9-2d02-41ee-9d9a-2f6cabff0970\") " pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-89jzw" Oct 30 22:02:02.323346 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:02.323312 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlc8f\" (UniqueName: \"kubernetes.io/projected/865819b9-2d02-41ee-9d9a-2f6cabff0970-kube-api-access-vlc8f\") pod \"e2e-praxis-a-71f39e-openshift-default-64c95f9f85-89jzw\" (UID: \"865819b9-2d02-41ee-9d9a-2f6cabff0970\") " pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-89jzw" Oct 30 22:02:02.324129 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:02.324107 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/865819b9-2d02-41ee-9d9a-2f6cabff0970-istio-token\") pod \"e2e-praxis-a-71f39e-openshift-default-64c95f9f85-89jzw\" (UID: \"865819b9-2d02-41ee-9d9a-2f6cabff0970\") " pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-89jzw" Oct 30 22:02:02.418215 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:02.416034 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:02.484671 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:02.484645 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-89jzw" Oct 30 22:02:02.777631 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:02.777605 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-89jzw"] Oct 30 22:02:02.778769 ip-10-0-137-181 kubenswrapper[6720]: W1030 22:02:02.778748 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod865819b9_2d02_41ee_9d9a_2f6cabff0970.slice/crio-a919e9832cb2e8392bca6b89d6ddab99f1a3e193723aa03a75da8db6d91477fa WatchSource:0}: Error finding container a919e9832cb2e8392bca6b89d6ddab99f1a3e193723aa03a75da8db6d91477fa: Status 404 returned error can't find the container with id a919e9832cb2e8392bca6b89d6ddab99f1a3e193723aa03a75da8db6d91477fa Oct 30 22:02:02.793728 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:02.793693 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:02:02.793835 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:02.793761 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:02:02.793835 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:02.793790 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:02:03.023759 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:03.023694 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:03.308245 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:03.308200 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:03.669820 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:03.665806 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:03.778092 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:03.778051 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-89jzw" event={"ID":"865819b9-2d02-41ee-9d9a-2f6cabff0970","Type":"ContainerStarted","Data":"1c16f782f6e643ed2eb06398f13ed22d88fc77e512753fd47077fb0ae4eaf75f"} Oct 30 22:02:03.778283 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:03.778116 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-89jzw" event={"ID":"865819b9-2d02-41ee-9d9a-2f6cabff0970","Type":"ContainerStarted","Data":"a919e9832cb2e8392bca6b89d6ddab99f1a3e193723aa03a75da8db6d91477fa"} Oct 30 22:02:03.821593 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:03.821534 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-89jzw" podStartSLOduration=1.82151931 podStartE2EDuration="1.82151931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:03.817517467 +0000 UTC m=+1289.988482649" watchObservedRunningTime="2026-10-30 22:02:03.82151931 +0000 UTC m=+1289.992484492" Oct 30 22:02:04.381187 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:04.381153 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:04.485834 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:04.485805 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-89jzw" Oct 30 22:02:04.487191 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:04.487156 6720 patch_prober.go:29] interesting pod/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-89jzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 22:02:04.487349 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:04.487220 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-89jzw" podUID="865819b9-2d02-41ee-9d9a-2f6cabff0970" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 22:02:04.652462 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:04.629231 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:05.101339 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:05.101263 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:05.485900 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:05.485868 6720 patch_prober.go:29] interesting pod/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-89jzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 22:02:05.486222 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:05.485917 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-89jzw" podUID="865819b9-2d02-41ee-9d9a-2f6cabff0970" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 22:02:05.821133 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:05.820741 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:05.996793 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:05.996749 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:06.485091 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:06.485059 6720 patch_prober.go:29] interesting pod/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-89jzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 22:02:06.485482 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:06.485121 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-89jzw" podUID="865819b9-2d02-41ee-9d9a-2f6cabff0970" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 22:02:06.641422 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:06.640633 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:07.428157 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:07.428107 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:07.485354 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:07.485321 6720 patch_prober.go:29] interesting pod/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-89jzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 22:02:07.485865 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:07.485370 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-89jzw" podUID="865819b9-2d02-41ee-9d9a-2f6cabff0970" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 22:02:08.191620 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:08.191519 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:08.486065 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:08.485984 6720 patch_prober.go:29] interesting pod/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-89jzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 22:02:08.486566 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:08.486540 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-89jzw" podUID="865819b9-2d02-41ee-9d9a-2f6cabff0970" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 22:02:09.051464 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:09.051310 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:09.211871 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:09.211831 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:09.491893 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:09.491864 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-89jzw" Oct 30 22:02:09.492368 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:09.492342 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-89jzw" Oct 30 22:02:09.493136 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:09.493117 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-89jzw" Oct 30 22:02:10.076488 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:10.076112 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:10.248950 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:10.248919 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:10.414563 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:02:10.414520 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 22:02:10.665001 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:10.664883 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:11.339367 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:11.339314 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:12.120345 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:12.120314 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:12.815758 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:12.815715 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:13.657482 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:13.657441 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:13.881950 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:13.881885 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:14.038686 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:14.038594 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:14.347596 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:14.347511 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:14.544680 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:14.544604 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:14.707448 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:14.707416 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:14.982320 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:14.982238 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:15.243136 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:15.243062 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:15.411367 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:15.411327 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:15.762603 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:15.762560 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:16.444320 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:16.444250 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:18.118850 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:18.118797 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:18.905513 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:18.905475 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:19.093484 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:19.093449 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:19.755061 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:19.755017 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:20.738433 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:20.738367 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:21.672570 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:21.672531 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:22.565371 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:22.565333 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:22.722040 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:22.722003 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:22.903334 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:22.903302 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:23.140408 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:23.138862 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:23.360544 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:23.354932 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:23.535887 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:23.535838 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:23.713137 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:23.713066 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:23.923774 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:23.923741 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:24.246178 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:24.246148 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:24.932061 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:24.932025 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:26.000564 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:26.000529 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:26.607493 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:26.607458 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:26.771226 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:26.771190 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:26.930354 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:26.922479 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-gxcgh"] Oct 30 22:02:26.930354 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:26.922807 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-gxcgh" podUID="0c73aea8-611d-4762-b2d2-6528eeb7dce9" containerName="istio-proxy" containerID="cri-o://152ea05b537bd0851a95d5a79703a6306ef729a7bae644977b7b61ecdd88327c" gracePeriod=30 Oct 30 22:02:26.968792 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:26.968757 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:27.136958 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:27.136913 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:27.483121 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:27.483092 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:27.934252 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:27.934215 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:28.316267 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:28.316165 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:28.815637 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:28.815607 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:29.588548 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:29.588514 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:29.812712 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:29.812677 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:30.142172 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:30.142134 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:30.413306 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:30.413216 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:30.856330 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:30.854907 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:31.513978 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:31.513930 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:32.219181 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:32.219143 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:32.256649 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:32.256614 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-gxcgh" Oct 30 22:02:32.315401 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:32.315348 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c73aea8-611d-4762-b2d2-6528eeb7dce9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c73aea8-611d-4762-b2d2-6528eeb7dce9-istio-envoy\") pod \"0c73aea8-611d-4762-b2d2-6528eeb7dce9\" (UID: \"0c73aea8-611d-4762-b2d2-6528eeb7dce9\") " Oct 30 22:02:32.315593 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:32.315470 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c73aea8-611d-4762-b2d2-6528eeb7dce9-istio-token\" (UniqueName: \"kubernetes.io/projected/0c73aea8-611d-4762-b2d2-6528eeb7dce9-istio-token\") pod \"0c73aea8-611d-4762-b2d2-6528eeb7dce9\" (UID: \"0c73aea8-611d-4762-b2d2-6528eeb7dce9\") " Oct 30 22:02:32.315593 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:32.315581 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c73aea8-611d-4762-b2d2-6528eeb7dce9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c73aea8-611d-4762-b2d2-6528eeb7dce9-credential-socket\") pod \"0c73aea8-611d-4762-b2d2-6528eeb7dce9\" (UID: \"0c73aea8-611d-4762-b2d2-6528eeb7dce9\") " Oct 30 22:02:32.315722 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:32.315706 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c73aea8-611d-4762-b2d2-6528eeb7dce9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c73aea8-611d-4762-b2d2-6528eeb7dce9-workload-socket\") pod \"0c73aea8-611d-4762-b2d2-6528eeb7dce9\" (UID: \"0c73aea8-611d-4762-b2d2-6528eeb7dce9\") " Oct 30 22:02:32.315777 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:32.315763 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c73aea8-611d-4762-b2d2-6528eeb7dce9-kube-api-access-d5j29\" (UniqueName: \"kubernetes.io/projected/0c73aea8-611d-4762-b2d2-6528eeb7dce9-kube-api-access-d5j29\") pod \"0c73aea8-611d-4762-b2d2-6528eeb7dce9\" (UID: \"0c73aea8-611d-4762-b2d2-6528eeb7dce9\") " Oct 30 22:02:32.315905 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:32.315885 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0c73aea8-611d-4762-b2d2-6528eeb7dce9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c73aea8-611d-4762-b2d2-6528eeb7dce9-istio-podinfo\") pod \"0c73aea8-611d-4762-b2d2-6528eeb7dce9\" (UID: \"0c73aea8-611d-4762-b2d2-6528eeb7dce9\") " Oct 30 22:02:32.315959 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:32.315928 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c73aea8-611d-4762-b2d2-6528eeb7dce9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c73aea8-611d-4762-b2d2-6528eeb7dce9-workload-certs\") pod \"0c73aea8-611d-4762-b2d2-6528eeb7dce9\" (UID: \"0c73aea8-611d-4762-b2d2-6528eeb7dce9\") " Oct 30 22:02:32.316016 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:32.315946 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c73aea8-611d-4762-b2d2-6528eeb7dce9-credential-socket" pod "0c73aea8-611d-4762-b2d2-6528eeb7dce9" (UID: "0c73aea8-611d-4762-b2d2-6528eeb7dce9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:32.316016 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:32.315981 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c73aea8-611d-4762-b2d2-6528eeb7dce9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c73aea8-611d-4762-b2d2-6528eeb7dce9-istiod-ca-cert\") pod \"0c73aea8-611d-4762-b2d2-6528eeb7dce9\" (UID: \"0c73aea8-611d-4762-b2d2-6528eeb7dce9\") " Oct 30 22:02:32.316187 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:32.316129 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c73aea8-611d-4762-b2d2-6528eeb7dce9-workload-socket" pod "0c73aea8-611d-4762-b2d2-6528eeb7dce9" (UID: "0c73aea8-611d-4762-b2d2-6528eeb7dce9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:32.316325 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:32.316218 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c73aea8-611d-4762-b2d2-6528eeb7dce9-workload-certs" pod "0c73aea8-611d-4762-b2d2-6528eeb7dce9" (UID: "0c73aea8-611d-4762-b2d2-6528eeb7dce9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:32.316325 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:32.316287 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c73aea8-611d-4762-b2d2-6528eeb7dce9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c73aea8-611d-4762-b2d2-6528eeb7dce9-istio-data\") pod \"0c73aea8-611d-4762-b2d2-6528eeb7dce9\" (UID: \"0c73aea8-611d-4762-b2d2-6528eeb7dce9\") " Oct 30 22:02:32.316955 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:32.316933 6720 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c73aea8-611d-4762-b2d2-6528eeb7dce9-workload-certs\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:32.317051 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:32.316959 6720 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c73aea8-611d-4762-b2d2-6528eeb7dce9-credential-socket\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:32.317051 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:32.316973 6720 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c73aea8-611d-4762-b2d2-6528eeb7dce9-workload-socket\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:32.317365 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:32.317345 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c73aea8-611d-4762-b2d2-6528eeb7dce9-istiod-ca-cert" pod "0c73aea8-611d-4762-b2d2-6528eeb7dce9" (UID: "0c73aea8-611d-4762-b2d2-6528eeb7dce9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:02:32.317505 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:32.317425 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c73aea8-611d-4762-b2d2-6528eeb7dce9-istio-data" pod "0c73aea8-611d-4762-b2d2-6528eeb7dce9" (UID: "0c73aea8-611d-4762-b2d2-6528eeb7dce9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:32.322480 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:32.322343 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c73aea8-611d-4762-b2d2-6528eeb7dce9-istio-token" pod "0c73aea8-611d-4762-b2d2-6528eeb7dce9" (UID: "0c73aea8-611d-4762-b2d2-6528eeb7dce9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:32.322762 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:32.322696 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c73aea8-611d-4762-b2d2-6528eeb7dce9-kube-api-access-d5j29" pod "0c73aea8-611d-4762-b2d2-6528eeb7dce9" (UID: "0c73aea8-611d-4762-b2d2-6528eeb7dce9"). InnerVolumeSpecName "kube-api-access-d5j29". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:32.323955 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:32.323933 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c73aea8-611d-4762-b2d2-6528eeb7dce9-istio-envoy" pod "0c73aea8-611d-4762-b2d2-6528eeb7dce9" (UID: "0c73aea8-611d-4762-b2d2-6528eeb7dce9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:32.324237 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:32.324214 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0c73aea8-611d-4762-b2d2-6528eeb7dce9-istio-podinfo" pod "0c73aea8-611d-4762-b2d2-6528eeb7dce9" (UID: "0c73aea8-611d-4762-b2d2-6528eeb7dce9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:02:32.386797 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:32.386734 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:32.418325 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:32.418290 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5j29\" (UniqueName: \"kubernetes.io/projected/0c73aea8-611d-4762-b2d2-6528eeb7dce9-kube-api-access-d5j29\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:32.418596 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:32.418330 6720 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c73aea8-611d-4762-b2d2-6528eeb7dce9-istio-podinfo\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:32.418596 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:32.418349 6720 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c73aea8-611d-4762-b2d2-6528eeb7dce9-istiod-ca-cert\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:32.418596 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:32.418362 6720 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c73aea8-611d-4762-b2d2-6528eeb7dce9-istio-data\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:32.418596 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:32.418374 6720 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c73aea8-611d-4762-b2d2-6528eeb7dce9-istio-envoy\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:32.418596 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:32.418402 6720 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c73aea8-611d-4762-b2d2-6528eeb7dce9-istio-token\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:32.620493 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:32.618014 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:32.677442 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:32.623919 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-89jzw"] Oct 30 22:02:32.688412 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:32.677909 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-89jzw" podUID="865819b9-2d02-41ee-9d9a-2f6cabff0970" containerName="istio-proxy" containerID="cri-o://1c16f782f6e643ed2eb06398f13ed22d88fc77e512753fd47077fb0ae4eaf75f" gracePeriod=30 Oct 30 22:02:32.772557 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:32.772522 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:33.025301 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:33.025220 6720 generic.go:356] "Generic (PLEG): container finished" podID="0c73aea8-611d-4762-b2d2-6528eeb7dce9" containerID="152ea05b537bd0851a95d5a79703a6306ef729a7bae644977b7b61ecdd88327c" exitCode=0 Oct 30 22:02:33.025301 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:33.025277 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-gxcgh" event={"ID":"0c73aea8-611d-4762-b2d2-6528eeb7dce9","Type":"ContainerDied","Data":"152ea05b537bd0851a95d5a79703a6306ef729a7bae644977b7b61ecdd88327c"} Oct 30 22:02:33.025547 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:33.025329 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-gxcgh" event={"ID":"0c73aea8-611d-4762-b2d2-6528eeb7dce9","Type":"ContainerDied","Data":"4e587fcdee791b417719c4de40503562cdf736d94929cb151606afd2585664ba"} Oct 30 22:02:33.025547 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:33.025285 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-gxcgh" Oct 30 22:02:33.025547 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:33.025356 6720 scope.go:122] "RemoveContainer" containerID="152ea05b537bd0851a95d5a79703a6306ef729a7bae644977b7b61ecdd88327c" Oct 30 22:02:33.036727 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:33.036707 6720 scope.go:122] "RemoveContainer" containerID="152ea05b537bd0851a95d5a79703a6306ef729a7bae644977b7b61ecdd88327c" Oct 30 22:02:33.037015 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:02:33.036995 6720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"152ea05b537bd0851a95d5a79703a6306ef729a7bae644977b7b61ecdd88327c\": container with ID starting with 152ea05b537bd0851a95d5a79703a6306ef729a7bae644977b7b61ecdd88327c not found: ID does not exist" containerID="152ea05b537bd0851a95d5a79703a6306ef729a7bae644977b7b61ecdd88327c" Oct 30 22:02:33.037081 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:33.037030 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"152ea05b537bd0851a95d5a79703a6306ef729a7bae644977b7b61ecdd88327c"} err="failed to get container status \"152ea05b537bd0851a95d5a79703a6306ef729a7bae644977b7b61ecdd88327c\": rpc error: code = NotFound desc = could not find container \"152ea05b537bd0851a95d5a79703a6306ef729a7bae644977b7b61ecdd88327c\": container with ID starting with 152ea05b537bd0851a95d5a79703a6306ef729a7bae644977b7b61ecdd88327c not found: ID does not exist" Oct 30 22:02:33.055769 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:33.055736 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-gxcgh"] Oct 30 22:02:33.062609 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:33.062580 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-3654d5-openshift-default-7d4bd67d65-gxcgh"] Oct 30 22:02:33.523479 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:33.523445 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:33.682406 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:33.680744 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:33.958255 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:33.958223 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c73aea8-611d-4762-b2d2-6528eeb7dce9" path="/var/lib/kubelet/pods/0c73aea8-611d-4762-b2d2-6528eeb7dce9/volumes" Oct 30 22:02:34.627832 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:34.627794 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:34.988613 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:34.988582 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:35.214412 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:35.203868 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:35.661888 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:35.661846 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:36.076353 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:36.076275 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:36.225227 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:36.225196 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:36.434188 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:36.434116 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:37.119516 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:37.119477 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:38.018866 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:38.018842 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-89jzw" Oct 30 22:02:38.021105 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:38.021067 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:38.060196 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:38.060166 6720 generic.go:356] "Generic (PLEG): container finished" podID="865819b9-2d02-41ee-9d9a-2f6cabff0970" containerID="1c16f782f6e643ed2eb06398f13ed22d88fc77e512753fd47077fb0ae4eaf75f" exitCode=0 Oct 30 22:02:38.060357 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:38.060224 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-89jzw" event={"ID":"865819b9-2d02-41ee-9d9a-2f6cabff0970","Type":"ContainerDied","Data":"1c16f782f6e643ed2eb06398f13ed22d88fc77e512753fd47077fb0ae4eaf75f"} Oct 30 22:02:38.060357 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:38.060228 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-89jzw" Oct 30 22:02:38.060357 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:38.060304 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-89jzw" event={"ID":"865819b9-2d02-41ee-9d9a-2f6cabff0970","Type":"ContainerDied","Data":"a919e9832cb2e8392bca6b89d6ddab99f1a3e193723aa03a75da8db6d91477fa"} Oct 30 22:02:38.060357 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:38.060330 6720 scope.go:122] "RemoveContainer" containerID="1c16f782f6e643ed2eb06398f13ed22d88fc77e512753fd47077fb0ae4eaf75f" Oct 30 22:02:38.075323 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:38.075278 6720 scope.go:122] "RemoveContainer" containerID="1c16f782f6e643ed2eb06398f13ed22d88fc77e512753fd47077fb0ae4eaf75f" Oct 30 22:02:38.075922 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:02:38.075896 6720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c16f782f6e643ed2eb06398f13ed22d88fc77e512753fd47077fb0ae4eaf75f\": container with ID starting with 1c16f782f6e643ed2eb06398f13ed22d88fc77e512753fd47077fb0ae4eaf75f not found: ID does not exist" containerID="1c16f782f6e643ed2eb06398f13ed22d88fc77e512753fd47077fb0ae4eaf75f" Oct 30 22:02:38.076094 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:38.075929 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c16f782f6e643ed2eb06398f13ed22d88fc77e512753fd47077fb0ae4eaf75f"} err="failed to get container status \"1c16f782f6e643ed2eb06398f13ed22d88fc77e512753fd47077fb0ae4eaf75f\": rpc error: code = NotFound desc = could not find container \"1c16f782f6e643ed2eb06398f13ed22d88fc77e512753fd47077fb0ae4eaf75f\": container with ID starting with 1c16f782f6e643ed2eb06398f13ed22d88fc77e512753fd47077fb0ae4eaf75f not found: ID does not exist" Oct 30 22:02:38.160574 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:38.160547 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/865819b9-2d02-41ee-9d9a-2f6cabff0970-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/865819b9-2d02-41ee-9d9a-2f6cabff0970-istio-data\") pod \"865819b9-2d02-41ee-9d9a-2f6cabff0970\" (UID: \"865819b9-2d02-41ee-9d9a-2f6cabff0970\") " Oct 30 22:02:38.161058 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:38.160590 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/865819b9-2d02-41ee-9d9a-2f6cabff0970-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/865819b9-2d02-41ee-9d9a-2f6cabff0970-istiod-ca-cert\") pod \"865819b9-2d02-41ee-9d9a-2f6cabff0970\" (UID: \"865819b9-2d02-41ee-9d9a-2f6cabff0970\") " Oct 30 22:02:38.161058 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:38.160683 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/865819b9-2d02-41ee-9d9a-2f6cabff0970-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/865819b9-2d02-41ee-9d9a-2f6cabff0970-workload-certs\") pod \"865819b9-2d02-41ee-9d9a-2f6cabff0970\" (UID: \"865819b9-2d02-41ee-9d9a-2f6cabff0970\") " Oct 30 22:02:38.161058 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:38.160712 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/865819b9-2d02-41ee-9d9a-2f6cabff0970-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/865819b9-2d02-41ee-9d9a-2f6cabff0970-istio-podinfo\") pod \"865819b9-2d02-41ee-9d9a-2f6cabff0970\" (UID: \"865819b9-2d02-41ee-9d9a-2f6cabff0970\") " Oct 30 22:02:38.161058 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:38.160731 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/865819b9-2d02-41ee-9d9a-2f6cabff0970-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/865819b9-2d02-41ee-9d9a-2f6cabff0970-workload-socket\") pod \"865819b9-2d02-41ee-9d9a-2f6cabff0970\" (UID: \"865819b9-2d02-41ee-9d9a-2f6cabff0970\") " Oct 30 22:02:38.161058 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:38.160752 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/865819b9-2d02-41ee-9d9a-2f6cabff0970-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/865819b9-2d02-41ee-9d9a-2f6cabff0970-credential-socket\") pod \"865819b9-2d02-41ee-9d9a-2f6cabff0970\" (UID: \"865819b9-2d02-41ee-9d9a-2f6cabff0970\") " Oct 30 22:02:38.161058 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:38.160800 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/865819b9-2d02-41ee-9d9a-2f6cabff0970-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/865819b9-2d02-41ee-9d9a-2f6cabff0970-istio-envoy\") pod \"865819b9-2d02-41ee-9d9a-2f6cabff0970\" (UID: \"865819b9-2d02-41ee-9d9a-2f6cabff0970\") " Oct 30 22:02:38.161058 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:38.160864 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/865819b9-2d02-41ee-9d9a-2f6cabff0970-istio-token\" (UniqueName: \"kubernetes.io/projected/865819b9-2d02-41ee-9d9a-2f6cabff0970-istio-token\") pod \"865819b9-2d02-41ee-9d9a-2f6cabff0970\" (UID: \"865819b9-2d02-41ee-9d9a-2f6cabff0970\") " Oct 30 22:02:38.161058 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:38.160872 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/865819b9-2d02-41ee-9d9a-2f6cabff0970-istio-data" pod "865819b9-2d02-41ee-9d9a-2f6cabff0970" (UID: "865819b9-2d02-41ee-9d9a-2f6cabff0970"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:38.161058 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:38.160965 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/865819b9-2d02-41ee-9d9a-2f6cabff0970-kube-api-access-vlc8f\" (UniqueName: \"kubernetes.io/projected/865819b9-2d02-41ee-9d9a-2f6cabff0970-kube-api-access-vlc8f\") pod \"865819b9-2d02-41ee-9d9a-2f6cabff0970\" (UID: \"865819b9-2d02-41ee-9d9a-2f6cabff0970\") " Oct 30 22:02:38.161778 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:38.161088 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/865819b9-2d02-41ee-9d9a-2f6cabff0970-workload-certs" pod "865819b9-2d02-41ee-9d9a-2f6cabff0970" (UID: "865819b9-2d02-41ee-9d9a-2f6cabff0970"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:38.161778 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:38.161153 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/865819b9-2d02-41ee-9d9a-2f6cabff0970-workload-socket" pod "865819b9-2d02-41ee-9d9a-2f6cabff0970" (UID: "865819b9-2d02-41ee-9d9a-2f6cabff0970"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:38.161778 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:38.161172 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/865819b9-2d02-41ee-9d9a-2f6cabff0970-istiod-ca-cert" pod "865819b9-2d02-41ee-9d9a-2f6cabff0970" (UID: "865819b9-2d02-41ee-9d9a-2f6cabff0970"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:02:38.161778 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:38.161311 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/865819b9-2d02-41ee-9d9a-2f6cabff0970-credential-socket" pod "865819b9-2d02-41ee-9d9a-2f6cabff0970" (UID: "865819b9-2d02-41ee-9d9a-2f6cabff0970"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:38.161778 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:38.161611 6720 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/865819b9-2d02-41ee-9d9a-2f6cabff0970-istio-data\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:38.161778 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:38.161635 6720 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/865819b9-2d02-41ee-9d9a-2f6cabff0970-istiod-ca-cert\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:38.161778 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:38.161650 6720 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/865819b9-2d02-41ee-9d9a-2f6cabff0970-workload-certs\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:38.161778 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:38.161665 6720 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/865819b9-2d02-41ee-9d9a-2f6cabff0970-workload-socket\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:38.161778 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:38.161677 6720 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/865819b9-2d02-41ee-9d9a-2f6cabff0970-credential-socket\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:38.163332 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:38.163308 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/865819b9-2d02-41ee-9d9a-2f6cabff0970-istio-envoy" pod "865819b9-2d02-41ee-9d9a-2f6cabff0970" (UID: "865819b9-2d02-41ee-9d9a-2f6cabff0970"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:02:38.163579 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:38.163558 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/865819b9-2d02-41ee-9d9a-2f6cabff0970-istio-podinfo" pod "865819b9-2d02-41ee-9d9a-2f6cabff0970" (UID: "865819b9-2d02-41ee-9d9a-2f6cabff0970"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:02:38.164056 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:38.164032 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/865819b9-2d02-41ee-9d9a-2f6cabff0970-istio-token" pod "865819b9-2d02-41ee-9d9a-2f6cabff0970" (UID: "865819b9-2d02-41ee-9d9a-2f6cabff0970"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:38.164056 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:38.164042 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/865819b9-2d02-41ee-9d9a-2f6cabff0970-kube-api-access-vlc8f" pod "865819b9-2d02-41ee-9d9a-2f6cabff0970" (UID: "865819b9-2d02-41ee-9d9a-2f6cabff0970"). InnerVolumeSpecName "kube-api-access-vlc8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:38.262576 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:38.262484 6720 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/865819b9-2d02-41ee-9d9a-2f6cabff0970-istio-podinfo\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:38.262576 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:38.262518 6720 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/865819b9-2d02-41ee-9d9a-2f6cabff0970-istio-envoy\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:38.262576 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:38.262532 6720 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/865819b9-2d02-41ee-9d9a-2f6cabff0970-istio-token\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:38.262576 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:38.262544 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlc8f\" (UniqueName: \"kubernetes.io/projected/865819b9-2d02-41ee-9d9a-2f6cabff0970-kube-api-access-vlc8f\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:38.384522 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:38.384488 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-89jzw"] Oct 30 22:02:38.389069 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:38.389046 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-71f39e-openshift-default-64c95f9f85-89jzw"] Oct 30 22:02:38.759672 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:38.759640 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:39.416462 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:39.410561 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:39.957905 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:39.957875 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="865819b9-2d02-41ee-9d9a-2f6cabff0970" path="/var/lib/kubelet/pods/865819b9-2d02-41ee-9d9a-2f6cabff0970/volumes" Oct 30 22:02:40.109093 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:40.109044 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:40.401978 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:02:40.401936 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 22:02:40.875411 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:40.871988 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:41.524570 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:41.524532 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:41.680858 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:41.680808 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:42.474378 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:42.474348 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:43.069814 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:43.069781 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:43.230443 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:43.226032 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:43.421899 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:43.421861 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:43.574284 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:43.574246 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:43.750921 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:43.750825 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:43.969342 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:43.969287 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:44.431005 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:44.430957 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:45.080421 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:45.079887 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:45.896119 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:45.896078 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:46.315979 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:46.315905 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e3670-6fb95fbcb8-zbsns"] Oct 30 22:02:46.316958 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:46.316924 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e3670-6fb95fbcb8-zbsns" podUID="907c3a08-898e-433a-8ec6-3ee3fceb949e" containerName="maas-api" containerID="cri-o://3140dad1fc7dbfe5027ff84bd52ebe10c573bc7af383a1cbdbfc27cfab042f85" gracePeriod=30 Oct 30 22:02:46.411819 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:46.411375 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:46.664411 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:46.650544 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:46.812543 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:46.809322 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:46.849187 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:46.849153 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e3670-6fb95fbcb8-zbsns" Oct 30 22:02:46.919505 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:46.915646 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/907c3a08-898e-433a-8ec6-3ee3fceb949e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/907c3a08-898e-433a-8ec6-3ee3fceb949e-openshift-service-ca\") pod \"907c3a08-898e-433a-8ec6-3ee3fceb949e\" (UID: \"907c3a08-898e-433a-8ec6-3ee3fceb949e\") " Oct 30 22:02:46.919505 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:46.916157 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/907c3a08-898e-433a-8ec6-3ee3fceb949e-openshift-service-ca" pod "907c3a08-898e-433a-8ec6-3ee3fceb949e" (UID: "907c3a08-898e-433a-8ec6-3ee3fceb949e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:02:46.919505 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:46.916999 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/907c3a08-898e-433a-8ec6-3ee3fceb949e-kube-api-access-pj9hq\" (UniqueName: \"kubernetes.io/projected/907c3a08-898e-433a-8ec6-3ee3fceb949e-kube-api-access-pj9hq\") pod \"907c3a08-898e-433a-8ec6-3ee3fceb949e\" (UID: \"907c3a08-898e-433a-8ec6-3ee3fceb949e\") " Oct 30 22:02:46.919505 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:46.917142 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/907c3a08-898e-433a-8ec6-3ee3fceb949e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/907c3a08-898e-433a-8ec6-3ee3fceb949e-maas-api-tls\") pod \"907c3a08-898e-433a-8ec6-3ee3fceb949e\" (UID: \"907c3a08-898e-433a-8ec6-3ee3fceb949e\") " Oct 30 22:02:46.919505 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:46.917195 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/907c3a08-898e-433a-8ec6-3ee3fceb949e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/907c3a08-898e-433a-8ec6-3ee3fceb949e-metrics-certs\") pod \"907c3a08-898e-433a-8ec6-3ee3fceb949e\" (UID: \"907c3a08-898e-433a-8ec6-3ee3fceb949e\") " Oct 30 22:02:46.919505 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:46.918644 6720 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/907c3a08-898e-433a-8ec6-3ee3fceb949e-openshift-service-ca\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:46.927135 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:46.926988 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/907c3a08-898e-433a-8ec6-3ee3fceb949e-maas-api-tls" pod "907c3a08-898e-433a-8ec6-3ee3fceb949e" (UID: "907c3a08-898e-433a-8ec6-3ee3fceb949e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:02:46.928473 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:46.928445 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/907c3a08-898e-433a-8ec6-3ee3fceb949e-metrics-certs" pod "907c3a08-898e-433a-8ec6-3ee3fceb949e" (UID: "907c3a08-898e-433a-8ec6-3ee3fceb949e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:02:46.932854 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:46.932832 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/907c3a08-898e-433a-8ec6-3ee3fceb949e-kube-api-access-pj9hq" pod "907c3a08-898e-433a-8ec6-3ee3fceb949e" (UID: "907c3a08-898e-433a-8ec6-3ee3fceb949e"). InnerVolumeSpecName "kube-api-access-pj9hq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:02:46.981798 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:46.981752 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:47.020833 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:47.020797 6720 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/907c3a08-898e-433a-8ec6-3ee3fceb949e-maas-api-tls\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:47.020833 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:47.020824 6720 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/907c3a08-898e-433a-8ec6-3ee3fceb949e-metrics-certs\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:47.020833 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:47.020835 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pj9hq\" (UniqueName: \"kubernetes.io/projected/907c3a08-898e-433a-8ec6-3ee3fceb949e-kube-api-access-pj9hq\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:02:47.147611 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:47.147571 6720 generic.go:356] "Generic (PLEG): container finished" podID="907c3a08-898e-433a-8ec6-3ee3fceb949e" containerID="3140dad1fc7dbfe5027ff84bd52ebe10c573bc7af383a1cbdbfc27cfab042f85" exitCode=0 Oct 30 22:02:47.147785 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:47.147636 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e3670-6fb95fbcb8-zbsns" event={"ID":"907c3a08-898e-433a-8ec6-3ee3fceb949e","Type":"ContainerDied","Data":"3140dad1fc7dbfe5027ff84bd52ebe10c573bc7af383a1cbdbfc27cfab042f85"} Oct 30 22:02:47.147785 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:47.147770 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e3670-6fb95fbcb8-zbsns" event={"ID":"907c3a08-898e-433a-8ec6-3ee3fceb949e","Type":"ContainerDied","Data":"abfa203ad20248dffa180ae11601fa0c13f7225de8165db0106597ad946bd0c5"} Oct 30 22:02:47.147785 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:47.147820 6720 scope.go:122] "RemoveContainer" containerID="3140dad1fc7dbfe5027ff84bd52ebe10c573bc7af383a1cbdbfc27cfab042f85" Oct 30 22:02:47.155144 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:47.155107 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e3670-6fb95fbcb8-zbsns" Oct 30 22:02:47.160101 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:47.160085 6720 scope.go:122] "RemoveContainer" containerID="3140dad1fc7dbfe5027ff84bd52ebe10c573bc7af383a1cbdbfc27cfab042f85" Oct 30 22:02:47.160473 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:02:47.160441 6720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3140dad1fc7dbfe5027ff84bd52ebe10c573bc7af383a1cbdbfc27cfab042f85\": container with ID starting with 3140dad1fc7dbfe5027ff84bd52ebe10c573bc7af383a1cbdbfc27cfab042f85 not found: ID does not exist" containerID="3140dad1fc7dbfe5027ff84bd52ebe10c573bc7af383a1cbdbfc27cfab042f85" Oct 30 22:02:47.160584 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:47.160484 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3140dad1fc7dbfe5027ff84bd52ebe10c573bc7af383a1cbdbfc27cfab042f85"} err="failed to get container status \"3140dad1fc7dbfe5027ff84bd52ebe10c573bc7af383a1cbdbfc27cfab042f85\": rpc error: code = NotFound desc = could not find container \"3140dad1fc7dbfe5027ff84bd52ebe10c573bc7af383a1cbdbfc27cfab042f85\": container with ID starting with 3140dad1fc7dbfe5027ff84bd52ebe10c573bc7af383a1cbdbfc27cfab042f85 not found: ID does not exist" Oct 30 22:02:47.187885 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:47.187814 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e3670-6fb95fbcb8-zbsns"] Oct 30 22:02:47.196421 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:47.196378 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-3e3670-6fb95fbcb8-zbsns"] Oct 30 22:02:47.327063 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:47.327028 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:47.950418 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:47.946687 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:47.962437 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:47.962373 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="907c3a08-898e-433a-8ec6-3ee3fceb949e" path="/var/lib/kubelet/pods/907c3a08-898e-433a-8ec6-3ee3fceb949e/volumes" Oct 30 22:02:48.472836 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:48.472751 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:48.628551 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:48.626728 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:48.783978 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:48.783888 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:48.987089 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:48.987062 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:49.291876 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:49.291837 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:49.756067 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:49.756033 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:50.731410 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:50.728120 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:51.922114 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:51.922069 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:52.146087 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:52.143292 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:52.446411 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:52.446315 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:53.336207 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:53.335395 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:53.651326 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:53.651284 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:54.296636 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:54.296602 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:54.884826 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:54.884795 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:55.065479 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:55.065441 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:55.226817 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:55.226735 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:55.481110 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:55.481024 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:55.917074 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:55.917031 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:56.552804 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:56.552767 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:56.871870 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:56.871839 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:57.396261 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:57.396211 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:57.573041 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:57.573005 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:57.754453 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:57.754328 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:58.586675 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:58.582838 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:02:59.453983 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:02:59.453938 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:00.426420 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:00.426373 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:01.226859 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:01.226825 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:01.413438 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:01.413368 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:01.569341 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:01.569264 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:02.098004 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:02.097968 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:02.784486 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:02.784450 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:03.051056 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:03.050932 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:03.379551 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:03.379512 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:04.108320 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:04.108286 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:04.929854 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:04.929797 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:05.337717 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:05.337616 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:05.493432 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:05.493344 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:05.790918 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:05.790876 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:06.536441 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:06.536409 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:07.207762 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:07.207707 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:07.870222 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:07.870188 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:08.013970 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:08.013936 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:08.175853 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:08.175764 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:08.799519 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:08.799489 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:09.460867 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:09.460832 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:10.004287 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:10.004244 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:10.153290 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:10.153257 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:10.310444 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:10.310327 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:10.507862 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:03:10.507825 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 22:03:10.698949 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:10.698901 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:13.484031 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:13.483909 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:13.773294 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:13.773194 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:14.246305 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:14.246273 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:15.792409 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:15.792331 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/distinct-w4-939a16-kserve-7f88f68c56-cngtk" podUID="85316c69-7ebd-4fb6-9195-44d1fba2dfbc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 22:03:16.295584 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:16.295527 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/simulator-w0-1529dc-kserve-8466d8d645-l54cf" podUID="257edb08-2e86-4f49-be57-848ace5958b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:16.296570 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:16.296541 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:16.297141 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:16.295453 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/distinct-w1-0f26c5-kserve-7d6b79498b-lvsjw" podUID="457d2f97-63ab-41f4-b761-294d0ba3045e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:16.668214 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:16.666490 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/premium-w3-ceb36a-kserve-6576bb476f-j5kdr" podUID="d065e788-c972-4683-84ed-86a70ed4791d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 22:03:16.668214 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:16.667219 6720 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ceb36a/premium-w3-ceb36a-kserve-6576bb476f-j5kdr" podUID="d065e788-c972-4683-84ed-86a70ed4791d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": context deadline exceeded" Oct 30 22:03:17.531279 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:17.531230 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:17.847014 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:17.846924 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:17.999196 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:17.999161 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:18.188519 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:18.188489 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:18.655184 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:18.655151 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:19.525966 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:19.525930 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:20.343622 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:20.343584 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:20.905459 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:20.905365 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:21.100712 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:21.100683 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:21.337941 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:21.337830 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:21.574232 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:21.574188 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:21.996860 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:21.996831 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:22.837199 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:22.837172 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:23.039993 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:23.039958 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:23.280106 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:23.280072 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:23.758017 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:23.757981 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:23.980738 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:23.980710 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:24.598524 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:24.598487 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:25.565465 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:25.565429 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:26.435401 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:26.435351 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:27.175738 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:27.175702 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:27.353813 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:27.353781 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:27.582952 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:27.582870 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:28.232137 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:28.232090 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:28.514666 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:28.514595 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:28.787114 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:28.787028 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:29.007899 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:29.007847 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:29.171326 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:29.171291 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:29.414745 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:29.414684 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:30.253295 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:30.253265 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:30.795093 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:30.795057 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:31.083005 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:31.082934 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:31.259963 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:31.259933 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:31.398204 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:31.398175 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:31.882497 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:31.882467 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:32.738488 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:32.738457 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:32.898029 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:32.897997 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:33.682626 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:33.682589 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:34.637289 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:34.637253 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:35.321547 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:35.321512 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:36.299787 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:36.299739 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:37.252522 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:37.252474 6720 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-939a16/distinct-w4-939a16-kserve-7f88f68c56-cngtk" podUID="85316c69-7ebd-4fb6-9195-44d1fba2dfbc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:37.542297 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:37.542200 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/premium-w3-ceb36a-kserve-6576bb476f-j5kdr" podUID="d065e788-c972-4683-84ed-86a70ed4791d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:37.594518 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:37.594473 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/simulator-w4-939a16-kserve-74f845bf84-4c7b6" podUID="9bb6805f-f355-4ea8-80b7-542215c42022" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:37.681410 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:37.680485 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-2-w3-ceb36a-kserve-7c59695467-46gdc" podUID="f1b6e92c-2952-4bcb-ae33-91e7271342fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:37.917513 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:37.917467 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/distinct-2-w1-0f26c5-kserve-74976f985c-fw28x" podUID="97e1e65d-1d50-4097-98ca-2944fa182807" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:38.096839 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:38.095881 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:38.274046 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:38.273961 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:38.464798 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:38.464759 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:39.021031 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:39.020997 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:40.011599 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:40.011560 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:40.414460 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:03:40.414419 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 22:03:40.768527 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:40.768436 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:41.426121 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:41.426083 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:42.197004 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:42.196971 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:42.930556 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:42.930510 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:43.713040 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:43.713000 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:44.468121 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:44.468091 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:45.112936 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:45.112897 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:45.859917 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:45.859874 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:46.587511 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:46.587441 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:47.266911 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:47.266866 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:47.524134 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:47.524008 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:48.124167 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:48.124133 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:48.806582 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:48.806551 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:49.505138 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:49.505101 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:50.120500 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:50.120444 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:50.809553 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:50.809514 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:51.536037 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:51.535952 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:52.245609 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:52.245551 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:52.397633 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:52.397584 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:53.084297 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:53.084256 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:54.062319 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:54.062286 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:54.739706 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:54.739631 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:54.899026 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:54.898989 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:55.762688 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:55.762655 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:57.082192 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:57.082144 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:03:58.788526 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:58.788475 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/distinct-w4-939a16-kserve-7f88f68c56-cngtk" podUID="85316c69-7ebd-4fb6-9195-44d1fba2dfbc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:58.965539 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:58.965484 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/distinct-2-w1-0f26c5-kserve-74976f985c-fw28x" podUID="97e1e65d-1d50-4097-98ca-2944fa182807" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:59.296527 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:59.296477 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/distinct-w1-0f26c5-kserve-7d6b79498b-lvsjw" podUID="457d2f97-63ab-41f4-b761-294d0ba3045e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:59.297938 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:59.297907 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/simulator-w0-1529dc-kserve-8466d8d645-l54cf" podUID="257edb08-2e86-4f49-be57-848ace5958b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:59.633414 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:59.631479 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-2-w3-ceb36a-kserve-7c59695467-46gdc" podUID="f1b6e92c-2952-4bcb-ae33-91e7271342fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:59.792828 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:59.790468 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/distinct-w4-939a16-kserve-7f88f68c56-cngtk" podUID="85316c69-7ebd-4fb6-9195-44d1fba2dfbc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:03:59.792828 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:03:59.791243 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-939a16/distinct-w4-939a16-kserve-7f88f68c56-cngtk" Oct 30 22:04:00.008514 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:00.003526 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/distinct-2-w1-0f26c5-kserve-74976f985c-fw28x" podUID="97e1e65d-1d50-4097-98ca-2944fa182807" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:00.008514 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:00.004353 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0f26c5/distinct-2-w1-0f26c5-kserve-74976f985c-fw28x" Oct 30 22:04:00.307477 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:00.300743 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/simulator-w0-1529dc-kserve-8466d8d645-l54cf" podUID="257edb08-2e86-4f49-be57-848ace5958b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:00.333413 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:00.306481 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/distinct-2-w0-1529dc-kserve-7788894c6-7mbrr" podUID="09662f19-1ef9-406c-85ef-e402505875b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:00.356692 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:00.310343 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1529dc/simulator-w0-1529dc-kserve-8466d8d645-l54cf" Oct 30 22:04:00.415521 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:00.415483 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:00.567473 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:00.545902 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/premium-w3-ceb36a-kserve-6576bb476f-j5kdr" podUID="d065e788-c972-4683-84ed-86a70ed4791d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:00.603164 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:00.603105 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/simulator-w4-939a16-kserve-74f845bf84-4c7b6" podUID="9bb6805f-f355-4ea8-80b7-542215c42022" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:00.637595 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:00.637541 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-2-w3-ceb36a-kserve-7c59695467-46gdc" podUID="f1b6e92c-2952-4bcb-ae33-91e7271342fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:00.658428 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:00.649836 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-2-w3-ceb36a-kserve-7c59695467-46gdc" Oct 30 22:04:00.863544 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:00.863490 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/distinct-w4-939a16-kserve-7f88f68c56-cngtk" podUID="85316c69-7ebd-4fb6-9195-44d1fba2dfbc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:01.012189 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:01.012149 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/distinct-2-w1-0f26c5-kserve-74976f985c-fw28x" podUID="97e1e65d-1d50-4097-98ca-2944fa182807" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 30 22:04:01.037372 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:01.037334 6720 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-0f26c5/distinct-2-w1-0f26c5-kserve-74976f985c-fw28x" podUID="97e1e65d-1d50-4097-98ca-2944fa182807" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": context deadline exceeded" Oct 30 22:04:01.099547 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:01.099503 6720 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:04:01.099896 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:01.099860 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl" podUID="63d87167-397f-476b-837f-e40c8f14a994" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:01.333454 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:01.313034 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/simulator-w0-1529dc-kserve-8466d8d645-l54cf" podUID="257edb08-2e86-4f49-be57-848ace5958b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 22:04:01.333454 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:01.313456 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/distinct-w1-0f26c5-kserve-7d6b79498b-lvsjw" podUID="457d2f97-63ab-41f4-b761-294d0ba3045e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 30 22:04:01.333454 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:01.314657 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-2-w3-ceb36a-kserve-7c59695467-46gdc" Oct 30 22:04:01.533452 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:01.533406 6720 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-rvzw9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:04:01.533646 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:01.533472 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-rvzw9" podUID="cc1b52ec-a38c-409e-810c-7ae3998c2ebd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:01.548679 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:01.548622 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/premium-w3-ceb36a-kserve-6576bb476f-j5kdr" podUID="d065e788-c972-4683-84ed-86a70ed4791d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:01.549142 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:01.549122 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ceb36a/premium-w3-ceb36a-kserve-6576bb476f-j5kdr" Oct 30 22:04:01.629924 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:01.629883 6720 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:04:01.630192 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:01.630168 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9" podUID="13913563-7222-407c-98b1-5c1a17d10b82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:01.703977 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:01.703948 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-939a16/distinct-w4-939a16-kserve-7f88f68c56-cngtk" Oct 30 22:04:01.704201 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:01.704190 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0f26c5/distinct-2-w1-0f26c5-kserve-74976f985c-fw28x" Oct 30 22:04:01.781798 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:01.781752 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1529dc/simulator-w0-1529dc-kserve-8466d8d645-l54cf" Oct 30 22:04:01.784496 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:01.784331 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ceb36a/premium-w3-ceb36a-kserve-6576bb476f-j5kdr" Oct 30 22:04:01.864439 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:01.862669 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:02.822899 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:02.822842 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:03.511365 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:03.511332 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:04.307520 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:04.307492 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:05.039724 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:05.039693 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:05.938521 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:05.938489 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:06.718372 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:06.718335 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:07.446999 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:07.446963 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:08.840753 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:08.840722 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:09.313533 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:09.313499 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:10.071286 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:10.071255 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:10.402890 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:04:10.402854 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 22:04:10.881453 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:10.881418 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:11.639108 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:11.639072 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:12.341319 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:12.340704 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:13.259266 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:13.259222 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:14.483164 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:14.483078 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:15.300612 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:15.300568 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/distinct-w1-0f26c5-kserve-7d6b79498b-lvsjw" podUID="457d2f97-63ab-41f4-b761-294d0ba3045e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:15.339432 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:15.339237 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-2-w3-ceb36a-kserve-7c59695467-46gdc" podUID="f1b6e92c-2952-4bcb-ae33-91e7271342fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 30 22:04:15.399528 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:15.399487 6720 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-2-w3-ceb36a-kserve-7c59695467-46gdc" podUID="f1b6e92c-2952-4bcb-ae33-91e7271342fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:16.088981 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:16.088932 6720 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:04:16.091526 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:16.091299 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl" podUID="63d87167-397f-476b-837f-e40c8f14a994" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:16.105407 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:16.103089 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:17.139002 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:17.138957 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:18.138638 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:18.138600 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:18.844755 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:18.844721 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:19.563152 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:19.563116 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:20.301098 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:20.300916 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:21.003358 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:21.003326 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:21.722435 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:21.719920 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:22.410939 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:22.410906 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:23.224447 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:23.224414 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:23.923332 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:23.923296 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:24.671898 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:24.671863 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:24.837601 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:24.837568 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:25.130016 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:25.129977 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:25.307294 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:25.307258 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:25.608855 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:25.608813 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:26.431982 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:26.431950 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:27.378017 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:27.377964 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-mvxk6"] Oct 30 22:04:27.378258 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:27.378230 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-mvxk6" podUID="8d0a4f90-fa48-4e0e-b6fa-167055c872f1" containerName="istio-proxy" containerID="cri-o://b523f2bb8ac7f1f156be990df6e687253d26332aa3b81c6df9589ddba2ebb712" gracePeriod=30 Oct 30 22:04:27.390767 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:27.390738 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:27.826047 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:27.825969 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:28.262766 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:28.262717 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:28.924100 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:28.924053 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:29.085768 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:29.085733 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:29.314726 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:29.314596 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:29.931085 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:29.931038 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:30.113407 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:30.112973 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:30.773515 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:30.773488 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:30.932635 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:30.932605 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:31.694937 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:31.694904 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:31.849420 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:31.849352 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:32.241277 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:32.241244 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:32.462568 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:32.462529 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:32.718881 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:32.718848 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-mvxk6" Oct 30 22:04:32.817289 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:32.817214 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d0a4f90-fa48-4e0e-b6fa-167055c872f1-istio-token\" (UniqueName: \"kubernetes.io/projected/8d0a4f90-fa48-4e0e-b6fa-167055c872f1-istio-token\") pod \"8d0a4f90-fa48-4e0e-b6fa-167055c872f1\" (UID: \"8d0a4f90-fa48-4e0e-b6fa-167055c872f1\") " Oct 30 22:04:32.817469 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:32.817314 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d0a4f90-fa48-4e0e-b6fa-167055c872f1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d0a4f90-fa48-4e0e-b6fa-167055c872f1-istio-envoy\") pod \"8d0a4f90-fa48-4e0e-b6fa-167055c872f1\" (UID: \"8d0a4f90-fa48-4e0e-b6fa-167055c872f1\") " Oct 30 22:04:32.817469 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:32.817374 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d0a4f90-fa48-4e0e-b6fa-167055c872f1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d0a4f90-fa48-4e0e-b6fa-167055c872f1-istiod-ca-cert\") pod \"8d0a4f90-fa48-4e0e-b6fa-167055c872f1\" (UID: \"8d0a4f90-fa48-4e0e-b6fa-167055c872f1\") " Oct 30 22:04:32.817598 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:32.817509 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d0a4f90-fa48-4e0e-b6fa-167055c872f1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d0a4f90-fa48-4e0e-b6fa-167055c872f1-istio-data\") pod \"8d0a4f90-fa48-4e0e-b6fa-167055c872f1\" (UID: \"8d0a4f90-fa48-4e0e-b6fa-167055c872f1\") " Oct 30 22:04:32.817654 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:32.817622 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d0a4f90-fa48-4e0e-b6fa-167055c872f1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d0a4f90-fa48-4e0e-b6fa-167055c872f1-workload-socket\") pod \"8d0a4f90-fa48-4e0e-b6fa-167055c872f1\" (UID: \"8d0a4f90-fa48-4e0e-b6fa-167055c872f1\") " Oct 30 22:04:32.817715 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:32.817666 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d0a4f90-fa48-4e0e-b6fa-167055c872f1-kube-api-access-n7mbx\" (UniqueName: \"kubernetes.io/projected/8d0a4f90-fa48-4e0e-b6fa-167055c872f1-kube-api-access-n7mbx\") pod \"8d0a4f90-fa48-4e0e-b6fa-167055c872f1\" (UID: \"8d0a4f90-fa48-4e0e-b6fa-167055c872f1\") " Oct 30 22:04:32.817769 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:32.817736 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d0a4f90-fa48-4e0e-b6fa-167055c872f1-istio-data" pod "8d0a4f90-fa48-4e0e-b6fa-167055c872f1" (UID: "8d0a4f90-fa48-4e0e-b6fa-167055c872f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:32.817769 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:32.817742 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d0a4f90-fa48-4e0e-b6fa-167055c872f1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d0a4f90-fa48-4e0e-b6fa-167055c872f1-workload-certs\") pod \"8d0a4f90-fa48-4e0e-b6fa-167055c872f1\" (UID: \"8d0a4f90-fa48-4e0e-b6fa-167055c872f1\") " Oct 30 22:04:32.817950 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:32.817854 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d0a4f90-fa48-4e0e-b6fa-167055c872f1-workload-socket" pod "8d0a4f90-fa48-4e0e-b6fa-167055c872f1" (UID: "8d0a4f90-fa48-4e0e-b6fa-167055c872f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:32.817950 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:32.817894 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d0a4f90-fa48-4e0e-b6fa-167055c872f1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d0a4f90-fa48-4e0e-b6fa-167055c872f1-credential-socket\") pod \"8d0a4f90-fa48-4e0e-b6fa-167055c872f1\" (UID: \"8d0a4f90-fa48-4e0e-b6fa-167055c872f1\") " Oct 30 22:04:32.817950 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:32.817898 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d0a4f90-fa48-4e0e-b6fa-167055c872f1-istiod-ca-cert" pod "8d0a4f90-fa48-4e0e-b6fa-167055c872f1" (UID: "8d0a4f90-fa48-4e0e-b6fa-167055c872f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:04:32.817950 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:32.817934 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d0a4f90-fa48-4e0e-b6fa-167055c872f1-workload-certs" pod "8d0a4f90-fa48-4e0e-b6fa-167055c872f1" (UID: "8d0a4f90-fa48-4e0e-b6fa-167055c872f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:32.818123 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:32.817939 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8d0a4f90-fa48-4e0e-b6fa-167055c872f1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d0a4f90-fa48-4e0e-b6fa-167055c872f1-istio-podinfo\") pod \"8d0a4f90-fa48-4e0e-b6fa-167055c872f1\" (UID: \"8d0a4f90-fa48-4e0e-b6fa-167055c872f1\") " Oct 30 22:04:32.818376 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:32.818299 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d0a4f90-fa48-4e0e-b6fa-167055c872f1-credential-socket" pod "8d0a4f90-fa48-4e0e-b6fa-167055c872f1" (UID: "8d0a4f90-fa48-4e0e-b6fa-167055c872f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:32.818945 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:32.818920 6720 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d0a4f90-fa48-4e0e-b6fa-167055c872f1-istiod-ca-cert\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:32.819048 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:32.818950 6720 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d0a4f90-fa48-4e0e-b6fa-167055c872f1-istio-data\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:32.819048 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:32.818966 6720 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d0a4f90-fa48-4e0e-b6fa-167055c872f1-workload-socket\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:32.819048 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:32.818980 6720 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d0a4f90-fa48-4e0e-b6fa-167055c872f1-workload-certs\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:32.819048 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:32.818995 6720 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d0a4f90-fa48-4e0e-b6fa-167055c872f1-credential-socket\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:32.820096 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:32.820070 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d0a4f90-fa48-4e0e-b6fa-167055c872f1-istio-token" pod "8d0a4f90-fa48-4e0e-b6fa-167055c872f1" (UID: "8d0a4f90-fa48-4e0e-b6fa-167055c872f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:32.820216 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:32.820191 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d0a4f90-fa48-4e0e-b6fa-167055c872f1-istio-envoy" pod "8d0a4f90-fa48-4e0e-b6fa-167055c872f1" (UID: "8d0a4f90-fa48-4e0e-b6fa-167055c872f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:04:32.820433 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:32.820371 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d0a4f90-fa48-4e0e-b6fa-167055c872f1-kube-api-access-n7mbx" pod "8d0a4f90-fa48-4e0e-b6fa-167055c872f1" (UID: "8d0a4f90-fa48-4e0e-b6fa-167055c872f1"). InnerVolumeSpecName "kube-api-access-n7mbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:04:32.821361 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:32.821338 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8d0a4f90-fa48-4e0e-b6fa-167055c872f1-istio-podinfo" pod "8d0a4f90-fa48-4e0e-b6fa-167055c872f1" (UID: "8d0a4f90-fa48-4e0e-b6fa-167055c872f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:04:32.854581 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:32.853166 6720 generic.go:356] "Generic (PLEG): container finished" podID="8d0a4f90-fa48-4e0e-b6fa-167055c872f1" containerID="b523f2bb8ac7f1f156be990df6e687253d26332aa3b81c6df9589ddba2ebb712" exitCode=0 Oct 30 22:04:32.854581 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:32.853371 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-mvxk6" event={"ID":"8d0a4f90-fa48-4e0e-b6fa-167055c872f1","Type":"ContainerDied","Data":"b523f2bb8ac7f1f156be990df6e687253d26332aa3b81c6df9589ddba2ebb712"} Oct 30 22:04:32.854581 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:32.853501 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-mvxk6" event={"ID":"8d0a4f90-fa48-4e0e-b6fa-167055c872f1","Type":"ContainerDied","Data":"d17cbc5bdc9665d671ac6947c8fea50e218d2c243cdc7b0f22eece2108e32d42"} Oct 30 22:04:32.854581 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:32.853534 6720 scope.go:122] "RemoveContainer" containerID="b523f2bb8ac7f1f156be990df6e687253d26332aa3b81c6df9589ddba2ebb712" Oct 30 22:04:32.854581 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:32.853733 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-mvxk6" Oct 30 22:04:32.861092 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:32.856764 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:32.878154 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:32.877900 6720 scope.go:122] "RemoveContainer" containerID="b523f2bb8ac7f1f156be990df6e687253d26332aa3b81c6df9589ddba2ebb712" Oct 30 22:04:32.878474 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:04:32.878451 6720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b523f2bb8ac7f1f156be990df6e687253d26332aa3b81c6df9589ddba2ebb712\": container with ID starting with b523f2bb8ac7f1f156be990df6e687253d26332aa3b81c6df9589ddba2ebb712 not found: ID does not exist" containerID="b523f2bb8ac7f1f156be990df6e687253d26332aa3b81c6df9589ddba2ebb712" Oct 30 22:04:32.878573 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:32.878483 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b523f2bb8ac7f1f156be990df6e687253d26332aa3b81c6df9589ddba2ebb712"} err="failed to get container status \"b523f2bb8ac7f1f156be990df6e687253d26332aa3b81c6df9589ddba2ebb712\": rpc error: code = NotFound desc = could not find container \"b523f2bb8ac7f1f156be990df6e687253d26332aa3b81c6df9589ddba2ebb712\": container with ID starting with b523f2bb8ac7f1f156be990df6e687253d26332aa3b81c6df9589ddba2ebb712 not found: ID does not exist" Oct 30 22:04:32.893495 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:32.891339 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-mvxk6"] Oct 30 22:04:32.901717 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:32.901687 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-mvxk6"] Oct 30 22:04:32.921281 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:32.921248 6720 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d0a4f90-fa48-4e0e-b6fa-167055c872f1-istio-podinfo\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:32.921450 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:32.921284 6720 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d0a4f90-fa48-4e0e-b6fa-167055c872f1-istio-token\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:32.921450 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:32.921300 6720 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d0a4f90-fa48-4e0e-b6fa-167055c872f1-istio-envoy\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:32.921450 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:32.921315 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7mbx\" (UniqueName: \"kubernetes.io/projected/8d0a4f90-fa48-4e0e-b6fa-167055c872f1-kube-api-access-n7mbx\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:04:33.000539 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:33.000502 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:33.166247 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:33.166210 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:33.970130 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:33.970094 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d0a4f90-fa48-4e0e-b6fa-167055c872f1" path="/var/lib/kubelet/pods/8d0a4f90-fa48-4e0e-b6fa-167055c872f1/volumes" Oct 30 22:04:34.004961 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:34.004932 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:34.368445 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:34.368398 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:34.543770 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:34.543724 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:35.050122 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:35.050086 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:35.757825 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:35.757797 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:35.946019 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:35.945979 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:36.336714 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:36.336677 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:36.612772 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:36.612580 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:37.237880 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:37.237845 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:37.417257 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:37.417228 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:38.035418 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:37.994877 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:38.565985 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:38.565946 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:38.987413 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:38.980494 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/premium-w3-ceb36a-kserve-6576bb476f-j5kdr" podUID="d065e788-c972-4683-84ed-86a70ed4791d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:39.014689 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:39.014632 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/distinct-2-w1-0f26c5-kserve-74976f985c-fw28x" podUID="97e1e65d-1d50-4097-98ca-2944fa182807" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:39.381578 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:39.381528 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-2-w3-ceb36a-kserve-7c59695467-46gdc" podUID="f1b6e92c-2952-4bcb-ae33-91e7271342fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:39.559532 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:39.559483 6720 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-1529dc/simulator-w0-1529dc-kserve-8466d8d645-l54cf" podUID="257edb08-2e86-4f49-be57-848ace5958b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:39.648462 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:39.646712 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/simulator-w4-939a16-kserve-74f845bf84-4c7b6" podUID="9bb6805f-f355-4ea8-80b7-542215c42022" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:39.980782 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:39.980688 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/premium-w3-ceb36a-kserve-6576bb476f-j5kdr" podUID="d065e788-c972-4683-84ed-86a70ed4791d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:40.068407 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:40.043447 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:40.069321 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:40.069286 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/distinct-2-w1-0f26c5-kserve-74976f985c-fw28x" podUID="97e1e65d-1d50-4097-98ca-2944fa182807" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 30 22:04:40.222413 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:40.190246 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ceb36a/premium-w3-ceb36a-kserve-6576bb476f-j5kdr" Oct 30 22:04:40.257376 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:40.257289 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0f26c5/distinct-2-w1-0f26c5-kserve-74976f985c-fw28x" Oct 30 22:04:40.304978 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:40.304468 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/distinct-w1-0f26c5-kserve-7d6b79498b-lvsjw" podUID="457d2f97-63ab-41f4-b761-294d0ba3045e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 30 22:04:40.348602 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:40.348540 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/simulator-w0-1529dc-kserve-8466d8d645-l54cf" podUID="257edb08-2e86-4f49-be57-848ace5958b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 22:04:40.356502 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:40.356464 6720 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-939a16/simulator-w4-939a16-kserve-74f845bf84-4c7b6" podUID="9bb6805f-f355-4ea8-80b7-542215c42022" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:40.467422 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:40.464472 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-2-w3-ceb36a-kserve-7c59695467-46gdc" podUID="f1b6e92c-2952-4bcb-ae33-91e7271342fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:40.467422 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:40.464890 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-2-w3-ceb36a-kserve-7c59695467-46gdc" Oct 30 22:04:40.650428 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:40.648530 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/simulator-w4-939a16-kserve-74f845bf84-4c7b6" podUID="9bb6805f-f355-4ea8-80b7-542215c42022" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:40.664547 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:40.655449 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-939a16/simulator-w4-939a16-kserve-74f845bf84-4c7b6" Oct 30 22:04:40.763419 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:04:40.751109 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 22:04:40.983749 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:40.981245 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/premium-w3-ceb36a-kserve-6576bb476f-j5kdr" podUID="d065e788-c972-4683-84ed-86a70ed4791d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:41.006688 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:40.998221 6720 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-0f26c5/distinct-2-w1-0f26c5-kserve-74976f985c-fw28x" podUID="97e1e65d-1d50-4097-98ca-2944fa182807" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": context deadline exceeded" Oct 30 22:04:41.063295 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:41.060486 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0f26c5/distinct-2-w1-0f26c5-kserve-74976f985c-fw28x" Oct 30 22:04:41.094881 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:41.093584 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-2-w3-ceb36a-kserve-7c59695467-46gdc" Oct 30 22:04:41.102655 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:41.100682 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ceb36a/premium-w3-ceb36a-kserve-6576bb476f-j5kdr" Oct 30 22:04:41.122441 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:41.122347 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-939a16/simulator-w4-939a16-kserve-74f845bf84-4c7b6" Oct 30 22:04:41.448643 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:41.448609 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:41.623901 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:41.623875 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:41.805343 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:41.805261 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:42.260436 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:42.256991 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:43.039326 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:43.039287 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:43.811049 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:43.810957 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:44.826298 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:44.826266 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:44.993664 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:44.993622 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:45.157886 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:45.157851 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:45.772485 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:45.772445 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:46.655304 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:46.655270 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:47.647685 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:47.647647 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5313f0-5d566fb777-cvvgv"] Oct 30 22:04:47.648579 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:47.648556 6720 cpu_manager.go:414] "containerMap: removing container" podUID="8d0a4f90-fa48-4e0e-b6fa-167055c872f1" containerName="istio-proxy" Oct 30 22:04:47.648694 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:47.648581 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d0a4f90-fa48-4e0e-b6fa-167055c872f1" containerName="istio-proxy" Oct 30 22:04:47.648694 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:47.648602 6720 cpu_manager.go:414] "containerMap: removing container" podUID="907c3a08-898e-433a-8ec6-3ee3fceb949e" containerName="maas-api" Oct 30 22:04:47.648694 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:47.648612 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="907c3a08-898e-433a-8ec6-3ee3fceb949e" containerName="maas-api" Oct 30 22:04:47.648694 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:47.648623 6720 cpu_manager.go:414] "containerMap: removing container" podUID="0c73aea8-611d-4762-b2d2-6528eeb7dce9" containerName="istio-proxy" Oct 30 22:04:47.648694 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:47.648630 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c73aea8-611d-4762-b2d2-6528eeb7dce9" containerName="istio-proxy" Oct 30 22:04:47.648694 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:47.648677 6720 cpu_manager.go:414] "containerMap: removing container" podUID="865819b9-2d02-41ee-9d9a-2f6cabff0970" containerName="istio-proxy" Oct 30 22:04:47.648694 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:47.648685 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="865819b9-2d02-41ee-9d9a-2f6cabff0970" containerName="istio-proxy" Oct 30 22:04:47.648929 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:47.648824 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="865819b9-2d02-41ee-9d9a-2f6cabff0970" containerName="istio-proxy" Oct 30 22:04:47.648929 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:47.648843 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d0a4f90-fa48-4e0e-b6fa-167055c872f1" containerName="istio-proxy" Oct 30 22:04:47.648929 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:47.648852 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="907c3a08-898e-433a-8ec6-3ee3fceb949e" containerName="maas-api" Oct 30 22:04:47.648929 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:47.648859 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c73aea8-611d-4762-b2d2-6528eeb7dce9" containerName="istio-proxy" Oct 30 22:04:47.654331 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:47.654307 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5313f0-5d566fb777-cvvgv" Oct 30 22:04:47.669140 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:47.669114 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-5313f0-serving-cert\"" Oct 30 22:04:47.680818 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:47.680791 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5313f0-5d566fb777-cvvgv"] Oct 30 22:04:47.748779 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:47.748745 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08f68597-808f-47ed-9878-b8ce40d02287-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-5313f0-5d566fb777-cvvgv\" (UID: \"08f68597-808f-47ed-9878-b8ce40d02287\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5313f0-5d566fb777-cvvgv" Oct 30 22:04:47.748945 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:47.748789 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08f68597-808f-47ed-9878-b8ce40d02287-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-5313f0-5d566fb777-cvvgv\" (UID: \"08f68597-808f-47ed-9878-b8ce40d02287\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5313f0-5d566fb777-cvvgv" Oct 30 22:04:47.749023 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:47.748998 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhjjv\" (UniqueName: \"kubernetes.io/projected/08f68597-808f-47ed-9878-b8ce40d02287-kube-api-access-nhjjv\") pod \"maas-api-e2e-shared-a-w2-5313f0-5d566fb777-cvvgv\" (UID: \"08f68597-808f-47ed-9878-b8ce40d02287\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5313f0-5d566fb777-cvvgv" Oct 30 22:04:47.749066 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:47.749053 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08f68597-808f-47ed-9878-b8ce40d02287-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-5313f0-5d566fb777-cvvgv\" (UID: \"08f68597-808f-47ed-9878-b8ce40d02287\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5313f0-5d566fb777-cvvgv" Oct 30 22:04:47.827160 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:47.827121 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:47.850009 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:47.849979 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08f68597-808f-47ed-9878-b8ce40d02287-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-5313f0-5d566fb777-cvvgv\" (UID: \"08f68597-808f-47ed-9878-b8ce40d02287\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5313f0-5d566fb777-cvvgv" Oct 30 22:04:47.850158 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:47.850126 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08f68597-808f-47ed-9878-b8ce40d02287-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-5313f0-5d566fb777-cvvgv\" (UID: \"08f68597-808f-47ed-9878-b8ce40d02287\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5313f0-5d566fb777-cvvgv" Oct 30 22:04:47.850214 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:47.850163 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08f68597-808f-47ed-9878-b8ce40d02287-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-5313f0-5d566fb777-cvvgv\" (UID: \"08f68597-808f-47ed-9878-b8ce40d02287\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5313f0-5d566fb777-cvvgv" Oct 30 22:04:47.850340 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:47.850323 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhjjv\" (UniqueName: \"kubernetes.io/projected/08f68597-808f-47ed-9878-b8ce40d02287-kube-api-access-nhjjv\") pod \"maas-api-e2e-shared-a-w2-5313f0-5d566fb777-cvvgv\" (UID: \"08f68597-808f-47ed-9878-b8ce40d02287\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5313f0-5d566fb777-cvvgv" Oct 30 22:04:47.850909 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:47.850845 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08f68597-808f-47ed-9878-b8ce40d02287-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-5313f0-5d566fb777-cvvgv\" (UID: \"08f68597-808f-47ed-9878-b8ce40d02287\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5313f0-5d566fb777-cvvgv" Oct 30 22:04:47.852768 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:47.852739 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08f68597-808f-47ed-9878-b8ce40d02287-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-5313f0-5d566fb777-cvvgv\" (UID: \"08f68597-808f-47ed-9878-b8ce40d02287\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5313f0-5d566fb777-cvvgv" Oct 30 22:04:47.852881 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:47.852784 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08f68597-808f-47ed-9878-b8ce40d02287-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-5313f0-5d566fb777-cvvgv\" (UID: \"08f68597-808f-47ed-9878-b8ce40d02287\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5313f0-5d566fb777-cvvgv" Oct 30 22:04:47.867406 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:47.867352 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhjjv\" (UniqueName: \"kubernetes.io/projected/08f68597-808f-47ed-9878-b8ce40d02287-kube-api-access-nhjjv\") pod \"maas-api-e2e-shared-a-w2-5313f0-5d566fb777-cvvgv\" (UID: \"08f68597-808f-47ed-9878-b8ce40d02287\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5313f0-5d566fb777-cvvgv" Oct 30 22:04:47.972874 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:47.972800 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5313f0-5d566fb777-cvvgv" Oct 30 22:04:47.981938 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:47.981889 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:48.329799 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:48.310642 6720 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:04:48.336430 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:48.335376 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5313f0-5d566fb777-cvvgv"] Oct 30 22:04:48.607751 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:48.607719 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:48.668797 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:48.668759 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5313f0-5d566fb777-cvvgv" event={"ID":"08f68597-808f-47ed-9878-b8ce40d02287","Type":"ContainerStarted","Data":"3d2094cd3f960a18ac70ca5fc53d78824becdfaaaf1f8b3f9bbd0535e357d1f1"} Oct 30 22:04:48.775287 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:48.775255 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:48.980141 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:48.980050 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:49.575108 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:49.575044 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:49.677657 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:49.677611 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5313f0-5d566fb777-cvvgv" event={"ID":"08f68597-808f-47ed-9878-b8ce40d02287","Type":"ContainerStarted","Data":"cbc0d384215c4473acf87700fde11799fc3ae3f9ddfbb4be530460ba6ba6094d"} Oct 30 22:04:49.677850 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:49.677681 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5313f0-5d566fb777-cvvgv" Oct 30 22:04:49.699897 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:49.699837 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5313f0-5d566fb777-cvvgv" podStartSLOduration=2.299778134 podStartE2EDuration="2.699818208s" totalImagesPullingTime="400.040074ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:49.698356935 +0000 UTC m=+1455.869322145" watchObservedRunningTime="2026-10-30 22:04:49.699818208 +0000 UTC m=+1455.870783386" Oct 30 22:04:50.346215 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:50.346184 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:51.299428 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:51.297562 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:52.219067 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:52.219030 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:52.903046 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:52.902998 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:53.524767 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:53.524731 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:54.109681 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:54.109654 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:54.256757 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:54.256728 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:54.558289 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:54.558174 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:54.863046 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:54.862998 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:55.011233 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:55.011195 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:55.469048 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:55.469013 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:55.691831 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:55.691794 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5313f0-5d566fb777-cvvgv" Oct 30 22:04:56.028014 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:56.027976 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:56.832932 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:56.832879 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:57.619944 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:57.619895 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:58.461776 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:58.461727 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:59.382326 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:59.382289 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:59.568191 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:59.568155 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:59.735521 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:59.735357 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:04:59.961242 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:04:59.961203 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:00.311811 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:00.311751 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:00.523873 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:00.523841 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:00.687412 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:00.682673 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:01.432008 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:01.431921 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:01.631281 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:01.631244 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:01.923116 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:01.923061 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:02.107414 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:02.106021 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:02.327607 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:02.327570 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:03.042409 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:03.041467 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:03.121429 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:03.120208 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/distinct-2-w1-0f26c5-kserve-74976f985c-fw28x" podUID="97e1e65d-1d50-4097-98ca-2944fa182807" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:03.121429 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:03.121330 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/premium-w3-ceb36a-kserve-6576bb476f-j5kdr" podUID="d065e788-c972-4683-84ed-86a70ed4791d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 22:05:03.121673 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:03.121505 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-2-w3-ceb36a-kserve-7c59695467-46gdc" podUID="f1b6e92c-2952-4bcb-ae33-91e7271342fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:03.132979 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:03.132942 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/simulator-w4-939a16-kserve-74f845bf84-4c7b6" podUID="9bb6805f-f355-4ea8-80b7-542215c42022" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:03.217854 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:03.217817 6720 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:05:03.218124 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:03.218095 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd" podUID="1817b106-b8fd-4989-977b-115eca7245ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:03.218403 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:03.218361 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:03.290595 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:03.290538 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/distinct-w1-0f26c5-kserve-7d6b79498b-lvsjw" podUID="457d2f97-63ab-41f4-b761-294d0ba3045e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:03.304682 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:03.304582 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/distinct-2-w0-1529dc-kserve-7788894c6-7mbrr" podUID="09662f19-1ef9-406c-85ef-e402505875b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:03.316696 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:03.316653 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/simulator-w0-1529dc-kserve-8466d8d645-l54cf" podUID="257edb08-2e86-4f49-be57-848ace5958b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:03.737408 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:03.716503 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/distinct-w4-939a16-kserve-7f88f68c56-cngtk" podUID="85316c69-7ebd-4fb6-9195-44d1fba2dfbc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:03.790176 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:03.790122 6720 patch_prober.go:29] interesting pod/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:05:03.790601 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:03.790533 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr" podUID="35de2b6b-4247-488f-baa1-182d56e82253" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:04.061464 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:04.060592 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:04.125409 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:04.124529 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/distinct-2-w1-0f26c5-kserve-74976f985c-fw28x" podUID="97e1e65d-1d50-4097-98ca-2944fa182807" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 30 22:05:04.125409 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:04.125316 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0f26c5/distinct-2-w1-0f26c5-kserve-74976f985c-fw28x" Oct 30 22:05:04.196412 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:04.171299 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-2-w3-ceb36a-kserve-7c59695467-46gdc" podUID="f1b6e92c-2952-4bcb-ae33-91e7271342fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 30 22:05:04.516477 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:04.196696 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-939a16/simulator-w4-939a16-kserve-74f845bf84-4c7b6" podUID="9bb6805f-f355-4ea8-80b7-542215c42022" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 22:05:04.565485 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:04.295514 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/distinct-w1-0f26c5-kserve-7d6b79498b-lvsjw" podUID="457d2f97-63ab-41f4-b761-294d0ba3045e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:04.565857 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:04.308948 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/distinct-2-w0-1529dc-kserve-7788894c6-7mbrr" podUID="09662f19-1ef9-406c-85ef-e402505875b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 22:05:04.566268 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:04.321146 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/simulator-w0-1529dc-kserve-8466d8d645-l54cf" podUID="257edb08-2e86-4f49-be57-848ace5958b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:05:04.579859 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:04.327500 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0f26c5/distinct-2-w1-0f26c5-kserve-74976f985c-fw28x" Oct 30 22:05:04.627807 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:04.627779 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-2-w3-ceb36a-kserve-7c59695467-46gdc" Oct 30 22:05:04.648481 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:04.648454 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-2-w3-ceb36a-kserve-7c59695467-46gdc" Oct 30 22:05:04.650423 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:04.650405 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-939a16/simulator-w4-939a16-kserve-74f845bf84-4c7b6" Oct 30 22:05:04.650675 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:04.650661 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-0f26c5/distinct-w1-0f26c5-kserve-7d6b79498b-lvsjw" Oct 30 22:05:04.694566 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:04.694537 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1529dc/distinct-2-w0-1529dc-kserve-7788894c6-7mbrr" Oct 30 22:05:04.802279 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:04.802202 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-1529dc/simulator-w0-1529dc-kserve-8466d8d645-l54cf" Oct 30 22:05:04.803332 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:04.803312 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-0f26c5/distinct-w1-0f26c5-kserve-7d6b79498b-lvsjw" Oct 30 22:05:04.804312 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:04.804298 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1529dc/distinct-2-w0-1529dc-kserve-7788894c6-7mbrr" Oct 30 22:05:04.804483 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:04.804471 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-1529dc/simulator-w0-1529dc-kserve-8466d8d645-l54cf" Oct 30 22:05:04.852329 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:04.852295 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-939a16/simulator-w4-939a16-kserve-74f845bf84-4c7b6" Oct 30 22:05:05.187437 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:05.186042 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:05.496951 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:05.496848 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:05.655228 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:05.655194 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:06.069080 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:06.069032 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:06.397155 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:06.397118 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:07.439668 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:07.439631 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:07.581712 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:07.581666 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:08.324441 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:08.324367 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:09.141600 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:09.141559 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:10.109192 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:10.109154 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:10.404201 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:05:10.404117 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 22:05:10.933913 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:10.933873 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:11.117090 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:11.117054 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:11.952866 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:11.952801 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:12.284939 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:12.284869 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:12.460158 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:12.460127 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:13.105549 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:13.105511 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-38d6e2-868dddd9c8-cv8wc"] Oct 30 22:05:13.117686 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:13.117657 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-38d6e2-868dddd9c8-cv8wc" Oct 30 22:05:13.124368 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:13.124337 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-38d6e2-868dddd9c8-cv8wc"] Oct 30 22:05:13.128154 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:13.128071 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-38d6e2-serving-cert\"" Oct 30 22:05:13.207348 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:13.207312 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/883a9007-3704-47d0-b59e-8591e1d33186-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-38d6e2-868dddd9c8-cv8wc\" (UID: \"883a9007-3704-47d0-b59e-8591e1d33186\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-38d6e2-868dddd9c8-cv8wc" Oct 30 22:05:13.207557 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:13.207443 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/883a9007-3704-47d0-b59e-8591e1d33186-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-38d6e2-868dddd9c8-cv8wc\" (UID: \"883a9007-3704-47d0-b59e-8591e1d33186\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-38d6e2-868dddd9c8-cv8wc" Oct 30 22:05:13.207615 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:13.207566 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzgcg\" (UniqueName: \"kubernetes.io/projected/883a9007-3704-47d0-b59e-8591e1d33186-kube-api-access-fzgcg\") pod \"maas-api-e2e-shared-b-w4-38d6e2-868dddd9c8-cv8wc\" (UID: \"883a9007-3704-47d0-b59e-8591e1d33186\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-38d6e2-868dddd9c8-cv8wc" Oct 30 22:05:13.207677 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:13.207661 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/883a9007-3704-47d0-b59e-8591e1d33186-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-38d6e2-868dddd9c8-cv8wc\" (UID: \"883a9007-3704-47d0-b59e-8591e1d33186\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-38d6e2-868dddd9c8-cv8wc" Oct 30 22:05:13.209599 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:13.209576 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:13.308888 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:13.308846 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/883a9007-3704-47d0-b59e-8591e1d33186-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-38d6e2-868dddd9c8-cv8wc\" (UID: \"883a9007-3704-47d0-b59e-8591e1d33186\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-38d6e2-868dddd9c8-cv8wc" Oct 30 22:05:13.309084 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:13.308956 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/883a9007-3704-47d0-b59e-8591e1d33186-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-38d6e2-868dddd9c8-cv8wc\" (UID: \"883a9007-3704-47d0-b59e-8591e1d33186\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-38d6e2-868dddd9c8-cv8wc" Oct 30 22:05:13.309084 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:13.309041 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzgcg\" (UniqueName: \"kubernetes.io/projected/883a9007-3704-47d0-b59e-8591e1d33186-kube-api-access-fzgcg\") pod \"maas-api-e2e-shared-b-w4-38d6e2-868dddd9c8-cv8wc\" (UID: \"883a9007-3704-47d0-b59e-8591e1d33186\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-38d6e2-868dddd9c8-cv8wc" Oct 30 22:05:13.309214 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:05:13.309083 6720 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-38d6e2-serving-cert: secret "maas-api-e2e-shared-b-w4-38d6e2-serving-cert" not found Oct 30 22:05:13.309214 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:13.309100 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/883a9007-3704-47d0-b59e-8591e1d33186-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-38d6e2-868dddd9c8-cv8wc\" (UID: \"883a9007-3704-47d0-b59e-8591e1d33186\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-38d6e2-868dddd9c8-cv8wc" Oct 30 22:05:13.309214 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:05:13.309173 6720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/883a9007-3704-47d0-b59e-8591e1d33186-maas-api-tls podName:883a9007-3704-47d0-b59e-8591e1d33186 nodeName:}" failed. No retries permitted until 2026-10-30 22:05:13.809149315 +0000 UTC m=+1479.980114493 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/883a9007-3704-47d0-b59e-8591e1d33186-maas-api-tls") pod "maas-api-e2e-shared-b-w4-38d6e2-868dddd9c8-cv8wc" (UID: "883a9007-3704-47d0-b59e-8591e1d33186") : secret "maas-api-e2e-shared-b-w4-38d6e2-serving-cert" not found Oct 30 22:05:13.310141 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:13.310112 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/883a9007-3704-47d0-b59e-8591e1d33186-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-38d6e2-868dddd9c8-cv8wc\" (UID: \"883a9007-3704-47d0-b59e-8591e1d33186\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-38d6e2-868dddd9c8-cv8wc" Oct 30 22:05:13.312303 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:13.312281 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/883a9007-3704-47d0-b59e-8591e1d33186-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-38d6e2-868dddd9c8-cv8wc\" (UID: \"883a9007-3704-47d0-b59e-8591e1d33186\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-38d6e2-868dddd9c8-cv8wc" Oct 30 22:05:13.332777 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:13.332737 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzgcg\" (UniqueName: \"kubernetes.io/projected/883a9007-3704-47d0-b59e-8591e1d33186-kube-api-access-fzgcg\") pod \"maas-api-e2e-shared-b-w4-38d6e2-868dddd9c8-cv8wc\" (UID: \"883a9007-3704-47d0-b59e-8591e1d33186\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-38d6e2-868dddd9c8-cv8wc" Oct 30 22:05:13.364482 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:13.364405 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:13.731816 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:13.731726 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:13.821409 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:13.817425 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/883a9007-3704-47d0-b59e-8591e1d33186-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-38d6e2-868dddd9c8-cv8wc\" (UID: \"883a9007-3704-47d0-b59e-8591e1d33186\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-38d6e2-868dddd9c8-cv8wc" Oct 30 22:05:13.821409 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:05:13.817697 6720 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-38d6e2-serving-cert: secret "maas-api-e2e-shared-b-w4-38d6e2-serving-cert" not found Oct 30 22:05:13.821409 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:05:13.817776 6720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/883a9007-3704-47d0-b59e-8591e1d33186-maas-api-tls podName:883a9007-3704-47d0-b59e-8591e1d33186 nodeName:}" failed. No retries permitted until 2026-10-30 22:05:14.8177581 +0000 UTC m=+1480.988723275 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/883a9007-3704-47d0-b59e-8591e1d33186-maas-api-tls") pod "maas-api-e2e-shared-b-w4-38d6e2-868dddd9c8-cv8wc" (UID: "883a9007-3704-47d0-b59e-8591e1d33186") : secret "maas-api-e2e-shared-b-w4-38d6e2-serving-cert" not found Oct 30 22:05:13.971427 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:13.971325 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:14.203742 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:14.203712 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:14.396794 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:14.396761 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:14.585367 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:14.585282 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:14.860636 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:14.860594 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/883a9007-3704-47d0-b59e-8591e1d33186-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-38d6e2-868dddd9c8-cv8wc\" (UID: \"883a9007-3704-47d0-b59e-8591e1d33186\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-38d6e2-868dddd9c8-cv8wc" Oct 30 22:05:14.863571 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:14.863548 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/883a9007-3704-47d0-b59e-8591e1d33186-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-38d6e2-868dddd9c8-cv8wc\" (UID: \"883a9007-3704-47d0-b59e-8591e1d33186\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-38d6e2-868dddd9c8-cv8wc" Oct 30 22:05:14.937730 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:14.937697 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-9j9cx"] Oct 30 22:05:14.937908 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:14.937773 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-38d6e2-868dddd9c8-cv8wc" Oct 30 22:05:14.943010 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:14.942704 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-9j9cx" Oct 30 22:05:14.964878 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:14.964845 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61ff25c3-567e-4fef-ad01-057108004032-istio-podinfo\") pod \"e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-9j9cx\" (UID: \"61ff25c3-567e-4fef-ad01-057108004032\") " pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-9j9cx" Oct 30 22:05:14.965105 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:14.965087 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61ff25c3-567e-4fef-ad01-057108004032-istio-data\") pod \"e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-9j9cx\" (UID: \"61ff25c3-567e-4fef-ad01-057108004032\") " pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-9j9cx" Oct 30 22:05:14.965275 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:14.965259 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61ff25c3-567e-4fef-ad01-057108004032-istiod-ca-cert\") pod \"e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-9j9cx\" (UID: \"61ff25c3-567e-4fef-ad01-057108004032\") " pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-9j9cx" Oct 30 22:05:14.965482 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:14.965466 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61ff25c3-567e-4fef-ad01-057108004032-istio-envoy\") pod \"e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-9j9cx\" (UID: \"61ff25c3-567e-4fef-ad01-057108004032\") " pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-9j9cx" Oct 30 22:05:14.965600 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:14.965586 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g5jm\" (UniqueName: \"kubernetes.io/projected/61ff25c3-567e-4fef-ad01-057108004032-kube-api-access-5g5jm\") pod \"e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-9j9cx\" (UID: \"61ff25c3-567e-4fef-ad01-057108004032\") " pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-9j9cx" Oct 30 22:05:14.966001 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:14.965979 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61ff25c3-567e-4fef-ad01-057108004032-workload-socket\") pod \"e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-9j9cx\" (UID: \"61ff25c3-567e-4fef-ad01-057108004032\") " pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-9j9cx" Oct 30 22:05:14.967185 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:14.967072 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61ff25c3-567e-4fef-ad01-057108004032-workload-certs\") pod \"e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-9j9cx\" (UID: \"61ff25c3-567e-4fef-ad01-057108004032\") " pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-9j9cx" Oct 30 22:05:14.967370 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:14.967351 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61ff25c3-567e-4fef-ad01-057108004032-istio-token\") pod \"e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-9j9cx\" (UID: \"61ff25c3-567e-4fef-ad01-057108004032\") " pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-9j9cx" Oct 30 22:05:14.967564 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:14.967548 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61ff25c3-567e-4fef-ad01-057108004032-credential-socket\") pod \"e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-9j9cx\" (UID: \"61ff25c3-567e-4fef-ad01-057108004032\") " pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-9j9cx" Oct 30 22:05:14.968176 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:14.968158 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-9j9cx"] Oct 30 22:05:15.038962 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:15.038921 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-cca1c7-openshift-default-dockercfg-lmtw8\"" Oct 30 22:05:15.068841 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:15.068816 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61ff25c3-567e-4fef-ad01-057108004032-workload-certs\") pod \"e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-9j9cx\" (UID: \"61ff25c3-567e-4fef-ad01-057108004032\") " pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-9j9cx" Oct 30 22:05:15.068979 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:15.068870 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61ff25c3-567e-4fef-ad01-057108004032-istio-token\") pod \"e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-9j9cx\" (UID: \"61ff25c3-567e-4fef-ad01-057108004032\") " pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-9j9cx" Oct 30 22:05:15.068979 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:15.068913 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61ff25c3-567e-4fef-ad01-057108004032-credential-socket\") pod \"e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-9j9cx\" (UID: \"61ff25c3-567e-4fef-ad01-057108004032\") " pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-9j9cx" Oct 30 22:05:15.068979 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:15.068947 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61ff25c3-567e-4fef-ad01-057108004032-istio-podinfo\") pod \"e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-9j9cx\" (UID: \"61ff25c3-567e-4fef-ad01-057108004032\") " pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-9j9cx" Oct 30 22:05:15.069170 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:15.068990 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61ff25c3-567e-4fef-ad01-057108004032-istio-data\") pod \"e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-9j9cx\" (UID: \"61ff25c3-567e-4fef-ad01-057108004032\") " pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-9j9cx" Oct 30 22:05:15.069170 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:15.069023 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61ff25c3-567e-4fef-ad01-057108004032-istiod-ca-cert\") pod \"e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-9j9cx\" (UID: \"61ff25c3-567e-4fef-ad01-057108004032\") " pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-9j9cx" Oct 30 22:05:15.069170 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:15.069127 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61ff25c3-567e-4fef-ad01-057108004032-istio-envoy\") pod \"e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-9j9cx\" (UID: \"61ff25c3-567e-4fef-ad01-057108004032\") " pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-9j9cx" Oct 30 22:05:15.069170 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:15.069154 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g5jm\" (UniqueName: \"kubernetes.io/projected/61ff25c3-567e-4fef-ad01-057108004032-kube-api-access-5g5jm\") pod \"e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-9j9cx\" (UID: \"61ff25c3-567e-4fef-ad01-057108004032\") " pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-9j9cx" Oct 30 22:05:15.069400 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:15.069211 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61ff25c3-567e-4fef-ad01-057108004032-workload-socket\") pod \"e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-9j9cx\" (UID: \"61ff25c3-567e-4fef-ad01-057108004032\") " pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-9j9cx" Oct 30 22:05:15.069400 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:15.069210 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61ff25c3-567e-4fef-ad01-057108004032-workload-certs\") pod \"e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-9j9cx\" (UID: \"61ff25c3-567e-4fef-ad01-057108004032\") " pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-9j9cx" Oct 30 22:05:15.069520 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:15.069466 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61ff25c3-567e-4fef-ad01-057108004032-istio-data\") pod \"e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-9j9cx\" (UID: \"61ff25c3-567e-4fef-ad01-057108004032\") " pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-9j9cx" Oct 30 22:05:15.070008 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:15.069985 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61ff25c3-567e-4fef-ad01-057108004032-istiod-ca-cert\") pod \"e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-9j9cx\" (UID: \"61ff25c3-567e-4fef-ad01-057108004032\") " pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-9j9cx" Oct 30 22:05:15.070597 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:15.070572 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61ff25c3-567e-4fef-ad01-057108004032-credential-socket\") pod \"e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-9j9cx\" (UID: \"61ff25c3-567e-4fef-ad01-057108004032\") " pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-9j9cx" Oct 30 22:05:15.070856 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:15.070836 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61ff25c3-567e-4fef-ad01-057108004032-workload-socket\") pod \"e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-9j9cx\" (UID: \"61ff25c3-567e-4fef-ad01-057108004032\") " pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-9j9cx" Oct 30 22:05:15.072672 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:15.072645 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61ff25c3-567e-4fef-ad01-057108004032-istio-envoy\") pod \"e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-9j9cx\" (UID: \"61ff25c3-567e-4fef-ad01-057108004032\") " pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-9j9cx" Oct 30 22:05:15.072822 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:15.072803 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61ff25c3-567e-4fef-ad01-057108004032-istio-podinfo\") pod \"e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-9j9cx\" (UID: \"61ff25c3-567e-4fef-ad01-057108004032\") " pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-9j9cx" Oct 30 22:05:15.090198 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:15.090163 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:15.090749 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:15.090719 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61ff25c3-567e-4fef-ad01-057108004032-istio-token\") pod \"e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-9j9cx\" (UID: \"61ff25c3-567e-4fef-ad01-057108004032\") " pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-9j9cx" Oct 30 22:05:15.092116 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:15.092086 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g5jm\" (UniqueName: \"kubernetes.io/projected/61ff25c3-567e-4fef-ad01-057108004032-kube-api-access-5g5jm\") pod \"e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-9j9cx\" (UID: \"61ff25c3-567e-4fef-ad01-057108004032\") " pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-9j9cx" Oct 30 22:05:15.302504 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:15.299759 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-9j9cx" Oct 30 22:05:15.312443 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:15.307728 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:15.345570 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:15.345537 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-38d6e2-868dddd9c8-cv8wc"] Oct 30 22:05:15.599663 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:15.599586 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:15.680337 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:15.680298 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-38d6e2-868dddd9c8-cv8wc" event={"ID":"883a9007-3704-47d0-b59e-8591e1d33186","Type":"ContainerStarted","Data":"d23f15d79da237aa7b378c1d4c95c472d12ce2f56bf6f96035b6fcc0cdd0234e"} Oct 30 22:05:15.717101 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:15.717078 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-9j9cx"] Oct 30 22:05:15.718164 ip-10-0-137-181 kubenswrapper[6720]: W1030 22:05:15.718136 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61ff25c3_567e_4fef_ad01_057108004032.slice/crio-edca9c7e640e7c9f6863b6f00156bb16de01cccefe922f00708d2aac61172234 WatchSource:0}: Error finding container edca9c7e640e7c9f6863b6f00156bb16de01cccefe922f00708d2aac61172234: Status 404 returned error can't find the container with id edca9c7e640e7c9f6863b6f00156bb16de01cccefe922f00708d2aac61172234 Oct 30 22:05:15.721642 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:15.721600 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:05:15.721769 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:15.721695 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:05:15.721769 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:15.721742 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:05:16.695409 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:16.689931 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-9j9cx" event={"ID":"61ff25c3-567e-4fef-ad01-057108004032","Type":"ContainerStarted","Data":"e71465d6b128647e6f6e729a3c7ecdac7f28111d79208f8935a7a64832b5fa0d"} Oct 30 22:05:16.695409 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:16.690030 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-9j9cx" event={"ID":"61ff25c3-567e-4fef-ad01-057108004032","Type":"ContainerStarted","Data":"edca9c7e640e7c9f6863b6f00156bb16de01cccefe922f00708d2aac61172234"} Oct 30 22:05:16.708829 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:16.707652 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-38d6e2-868dddd9c8-cv8wc" event={"ID":"883a9007-3704-47d0-b59e-8591e1d33186","Type":"ContainerStarted","Data":"f637de80f4276a149cf7718d95893d6e4cc9059d6c6f957167d69d4fd2171c0e"} Oct 30 22:05:16.708829 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:16.708644 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-38d6e2-868dddd9c8-cv8wc" Oct 30 22:05:16.734812 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:16.734622 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-9j9cx" podStartSLOduration=2.734605883 podStartE2EDuration="2.734605883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:16.732980056 +0000 UTC m=+1482.903945235" watchObservedRunningTime="2026-10-30 22:05:16.734605883 +0000 UTC m=+1482.905571063" Oct 30 22:05:16.743989 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:16.743960 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:17.308227 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:17.308193 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-9j9cx" Oct 30 22:05:17.309687 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:17.309663 6720 patch_prober.go:29] interesting pod/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-9j9cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 30 22:05:17.309833 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:17.309710 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-9j9cx" podUID="61ff25c3-567e-4fef-ad01-057108004032" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 30 22:05:17.630768 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:17.630731 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:18.308691 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:18.308659 6720 patch_prober.go:29] interesting pod/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-9j9cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 30 22:05:18.309155 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:18.308717 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-9j9cx" podUID="61ff25c3-567e-4fef-ad01-057108004032" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 30 22:05:18.580175 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:18.580080 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-38d6e2-868dddd9c8-cv8wc" podStartSLOduration=5.160239984 podStartE2EDuration="5.58006647s" totalImagesPullingTime="419.826486ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:16.779798727 +0000 UTC m=+1482.950763910" watchObservedRunningTime="2026-10-30 22:05:18.58006647 +0000 UTC m=+1484.751031680" Oct 30 22:05:18.584062 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:18.584017 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0f73c9-88bff4c9f-qd4gf"] Oct 30 22:05:18.592930 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:18.592902 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0f73c9-88bff4c9f-qd4gf"] Oct 30 22:05:18.593086 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:18.593015 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0f73c9-88bff4c9f-qd4gf" Oct 30 22:05:18.600091 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:18.600065 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-0f73c9-serving-cert\"" Oct 30 22:05:18.674938 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:18.674911 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:18.679559 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:18.679527 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f51bd7b2-4d7e-42af-9dbc-4083db247bf4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-0f73c9-88bff4c9f-qd4gf\" (UID: \"f51bd7b2-4d7e-42af-9dbc-4083db247bf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0f73c9-88bff4c9f-qd4gf" Oct 30 22:05:18.679698 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:18.679590 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f51bd7b2-4d7e-42af-9dbc-4083db247bf4-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-0f73c9-88bff4c9f-qd4gf\" (UID: \"f51bd7b2-4d7e-42af-9dbc-4083db247bf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0f73c9-88bff4c9f-qd4gf" Oct 30 22:05:18.679698 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:18.679687 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpwkn\" (UniqueName: \"kubernetes.io/projected/f51bd7b2-4d7e-42af-9dbc-4083db247bf4-kube-api-access-gpwkn\") pod \"maas-api-e2e-shared-b-w3-0f73c9-88bff4c9f-qd4gf\" (UID: \"f51bd7b2-4d7e-42af-9dbc-4083db247bf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0f73c9-88bff4c9f-qd4gf" Oct 30 22:05:18.679842 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:18.679821 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f51bd7b2-4d7e-42af-9dbc-4083db247bf4-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-0f73c9-88bff4c9f-qd4gf\" (UID: \"f51bd7b2-4d7e-42af-9dbc-4083db247bf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0f73c9-88bff4c9f-qd4gf" Oct 30 22:05:18.787444 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:18.785455 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f51bd7b2-4d7e-42af-9dbc-4083db247bf4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-0f73c9-88bff4c9f-qd4gf\" (UID: \"f51bd7b2-4d7e-42af-9dbc-4083db247bf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0f73c9-88bff4c9f-qd4gf" Oct 30 22:05:18.787444 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:18.785619 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f51bd7b2-4d7e-42af-9dbc-4083db247bf4-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-0f73c9-88bff4c9f-qd4gf\" (UID: \"f51bd7b2-4d7e-42af-9dbc-4083db247bf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0f73c9-88bff4c9f-qd4gf" Oct 30 22:05:18.787444 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:18.785695 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpwkn\" (UniqueName: \"kubernetes.io/projected/f51bd7b2-4d7e-42af-9dbc-4083db247bf4-kube-api-access-gpwkn\") pod \"maas-api-e2e-shared-b-w3-0f73c9-88bff4c9f-qd4gf\" (UID: \"f51bd7b2-4d7e-42af-9dbc-4083db247bf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0f73c9-88bff4c9f-qd4gf" Oct 30 22:05:18.787444 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:18.785884 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f51bd7b2-4d7e-42af-9dbc-4083db247bf4-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-0f73c9-88bff4c9f-qd4gf\" (UID: \"f51bd7b2-4d7e-42af-9dbc-4083db247bf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0f73c9-88bff4c9f-qd4gf" Oct 30 22:05:18.817406 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:18.797077 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f51bd7b2-4d7e-42af-9dbc-4083db247bf4-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-0f73c9-88bff4c9f-qd4gf\" (UID: \"f51bd7b2-4d7e-42af-9dbc-4083db247bf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0f73c9-88bff4c9f-qd4gf" Oct 30 22:05:18.817406 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:18.814319 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f51bd7b2-4d7e-42af-9dbc-4083db247bf4-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-0f73c9-88bff4c9f-qd4gf\" (UID: \"f51bd7b2-4d7e-42af-9dbc-4083db247bf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0f73c9-88bff4c9f-qd4gf" Oct 30 22:05:18.817406 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:18.814535 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f51bd7b2-4d7e-42af-9dbc-4083db247bf4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-0f73c9-88bff4c9f-qd4gf\" (UID: \"f51bd7b2-4d7e-42af-9dbc-4083db247bf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0f73c9-88bff4c9f-qd4gf" Oct 30 22:05:18.820919 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:18.820888 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpwkn\" (UniqueName: \"kubernetes.io/projected/f51bd7b2-4d7e-42af-9dbc-4083db247bf4-kube-api-access-gpwkn\") pod \"maas-api-e2e-shared-b-w3-0f73c9-88bff4c9f-qd4gf\" (UID: \"f51bd7b2-4d7e-42af-9dbc-4083db247bf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0f73c9-88bff4c9f-qd4gf" Oct 30 22:05:18.834336 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:18.834259 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:18.911725 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:18.911693 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0f73c9-88bff4c9f-qd4gf" Oct 30 22:05:19.320405 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:19.311461 6720 patch_prober.go:29] interesting pod/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-9j9cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 30 22:05:19.320405 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:19.311520 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-9j9cx" podUID="61ff25c3-567e-4fef-ad01-057108004032" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 30 22:05:19.392834 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:19.392800 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0f73c9-88bff4c9f-qd4gf"] Oct 30 22:05:19.400485 ip-10-0-137-181 kubenswrapper[6720]: W1030 22:05:19.400450 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf51bd7b2_4d7e_42af_9dbc_4083db247bf4.slice/crio-f7075771fc8b2ab6531751528821614944154144de9b42c4d7ec122327d97800 WatchSource:0}: Error finding container f7075771fc8b2ab6531751528821614944154144de9b42c4d7ec122327d97800: Status 404 returned error can't find the container with id f7075771fc8b2ab6531751528821614944154144de9b42c4d7ec122327d97800 Oct 30 22:05:19.731073 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:19.731006 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0f73c9-88bff4c9f-qd4gf" event={"ID":"f51bd7b2-4d7e-42af-9dbc-4083db247bf4","Type":"ContainerStarted","Data":"f7075771fc8b2ab6531751528821614944154144de9b42c4d7ec122327d97800"} Oct 30 22:05:19.733098 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:19.733064 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:20.308968 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:20.308882 6720 patch_prober.go:29] interesting pod/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-9j9cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 30 22:05:20.308968 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:20.308947 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-9j9cx" podUID="61ff25c3-567e-4fef-ad01-057108004032" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 30 22:05:20.542207 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:20.540924 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:20.743524 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:20.743490 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0f73c9-88bff4c9f-qd4gf" event={"ID":"f51bd7b2-4d7e-42af-9dbc-4083db247bf4","Type":"ContainerStarted","Data":"476330ecbd910a3b75ce1fc6afde42532023b9a4051aaea76c6e4eb032b213e5"} Oct 30 22:05:20.743730 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:20.743551 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0f73c9-88bff4c9f-qd4gf" Oct 30 22:05:20.777080 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:20.777027 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0f73c9-88bff4c9f-qd4gf" podStartSLOduration=2.278484389 podStartE2EDuration="2.777011728s" totalImagesPullingTime="498.527339ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:20.775472884 +0000 UTC m=+1486.946438070" watchObservedRunningTime="2026-10-30 22:05:20.777011728 +0000 UTC m=+1486.947976909" Oct 30 22:05:21.308249 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:21.308210 6720 patch_prober.go:29] interesting pod/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-9j9cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 30 22:05:21.308460 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:21.308276 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-9j9cx" podUID="61ff25c3-567e-4fef-ad01-057108004032" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 30 22:05:21.361539 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:21.359538 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:22.056666 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:22.056631 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:22.313563 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:22.313463 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-9j9cx" Oct 30 22:05:22.313799 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:22.313777 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-9j9cx" Oct 30 22:05:22.314453 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:22.314434 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-9j9cx" Oct 30 22:05:22.410231 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:22.410203 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:22.773938 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:22.773902 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:23.726844 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:23.726818 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-38d6e2-868dddd9c8-cv8wc" Oct 30 22:05:23.815645 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:23.815608 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0a2577-7f97948b44-d669v"] Oct 30 22:05:23.819981 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:23.819956 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0a2577-7f97948b44-d669v" Oct 30 22:05:23.829633 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:23.829605 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-0a2577-serving-cert\"" Oct 30 22:05:23.838247 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:23.838219 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0a2577-7f97948b44-d669v"] Oct 30 22:05:23.870911 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:23.870879 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7365fe23-d73c-467d-bd8b-d0177976452f-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-0a2577-7f97948b44-d669v\" (UID: \"7365fe23-d73c-467d-bd8b-d0177976452f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0a2577-7f97948b44-d669v" Oct 30 22:05:23.871113 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:23.870998 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7365fe23-d73c-467d-bd8b-d0177976452f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-0a2577-7f97948b44-d669v\" (UID: \"7365fe23-d73c-467d-bd8b-d0177976452f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0a2577-7f97948b44-d669v" Oct 30 22:05:23.871113 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:23.871099 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7365fe23-d73c-467d-bd8b-d0177976452f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-0a2577-7f97948b44-d669v\" (UID: \"7365fe23-d73c-467d-bd8b-d0177976452f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0a2577-7f97948b44-d669v" Oct 30 22:05:23.871264 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:23.871132 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bk56p\" (UniqueName: \"kubernetes.io/projected/7365fe23-d73c-467d-bd8b-d0177976452f-kube-api-access-bk56p\") pod \"maas-api-e2e-shared-b-w0-0a2577-7f97948b44-d669v\" (UID: \"7365fe23-d73c-467d-bd8b-d0177976452f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0a2577-7f97948b44-d669v" Oct 30 22:05:23.990996 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:23.980817 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7365fe23-d73c-467d-bd8b-d0177976452f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-0a2577-7f97948b44-d669v\" (UID: \"7365fe23-d73c-467d-bd8b-d0177976452f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0a2577-7f97948b44-d669v" Oct 30 22:05:23.990996 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:23.980882 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk56p\" (UniqueName: \"kubernetes.io/projected/7365fe23-d73c-467d-bd8b-d0177976452f-kube-api-access-bk56p\") pod \"maas-api-e2e-shared-b-w0-0a2577-7f97948b44-d669v\" (UID: \"7365fe23-d73c-467d-bd8b-d0177976452f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0a2577-7f97948b44-d669v" Oct 30 22:05:23.990996 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:23.981112 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7365fe23-d73c-467d-bd8b-d0177976452f-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-0a2577-7f97948b44-d669v\" (UID: \"7365fe23-d73c-467d-bd8b-d0177976452f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0a2577-7f97948b44-d669v" Oct 30 22:05:23.990996 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:23.981222 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7365fe23-d73c-467d-bd8b-d0177976452f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-0a2577-7f97948b44-d669v\" (UID: \"7365fe23-d73c-467d-bd8b-d0177976452f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0a2577-7f97948b44-d669v" Oct 30 22:05:23.990996 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:05:23.981406 6720 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0a2577-serving-cert: secret "maas-api-e2e-shared-b-w0-0a2577-serving-cert" not found Oct 30 22:05:23.990996 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:05:23.981497 6720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7365fe23-d73c-467d-bd8b-d0177976452f-maas-api-tls podName:7365fe23-d73c-467d-bd8b-d0177976452f nodeName:}" failed. No retries permitted until 2026-10-30 22:05:24.481481029 +0000 UTC m=+1490.652446188 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7365fe23-d73c-467d-bd8b-d0177976452f-maas-api-tls") pod "maas-api-e2e-shared-b-w0-0a2577-7f97948b44-d669v" (UID: "7365fe23-d73c-467d-bd8b-d0177976452f") : secret "maas-api-e2e-shared-b-w0-0a2577-serving-cert" not found Oct 30 22:05:23.990996 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:23.982423 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7365fe23-d73c-467d-bd8b-d0177976452f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-0a2577-7f97948b44-d669v\" (UID: \"7365fe23-d73c-467d-bd8b-d0177976452f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0a2577-7f97948b44-d669v" Oct 30 22:05:23.990996 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:23.984817 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:23.992215 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:23.992186 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7365fe23-d73c-467d-bd8b-d0177976452f-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-0a2577-7f97948b44-d669v\" (UID: \"7365fe23-d73c-467d-bd8b-d0177976452f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0a2577-7f97948b44-d669v" Oct 30 22:05:23.996120 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:23.996093 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk56p\" (UniqueName: \"kubernetes.io/projected/7365fe23-d73c-467d-bd8b-d0177976452f-kube-api-access-bk56p\") pod \"maas-api-e2e-shared-b-w0-0a2577-7f97948b44-d669v\" (UID: \"7365fe23-d73c-467d-bd8b-d0177976452f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0a2577-7f97948b44-d669v" Oct 30 22:05:24.170185 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:24.170151 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:24.493520 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:24.493320 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7365fe23-d73c-467d-bd8b-d0177976452f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-0a2577-7f97948b44-d669v\" (UID: \"7365fe23-d73c-467d-bd8b-d0177976452f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0a2577-7f97948b44-d669v" Oct 30 22:05:24.493520 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:05:24.493519 6720 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0a2577-serving-cert: secret "maas-api-e2e-shared-b-w0-0a2577-serving-cert" not found Oct 30 22:05:24.493780 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:05:24.493597 6720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7365fe23-d73c-467d-bd8b-d0177976452f-maas-api-tls podName:7365fe23-d73c-467d-bd8b-d0177976452f nodeName:}" failed. No retries permitted until 2026-10-30 22:05:25.493578407 +0000 UTC m=+1491.664543572 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7365fe23-d73c-467d-bd8b-d0177976452f-maas-api-tls") pod "maas-api-e2e-shared-b-w0-0a2577-7f97948b44-d669v" (UID: "7365fe23-d73c-467d-bd8b-d0177976452f") : secret "maas-api-e2e-shared-b-w0-0a2577-serving-cert" not found Oct 30 22:05:24.682476 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:24.682443 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:24.872778 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:24.872736 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:25.346054 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:25.345964 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:25.507752 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:25.507715 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7365fe23-d73c-467d-bd8b-d0177976452f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-0a2577-7f97948b44-d669v\" (UID: \"7365fe23-d73c-467d-bd8b-d0177976452f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0a2577-7f97948b44-d669v" Oct 30 22:05:25.510242 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:25.510219 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7365fe23-d73c-467d-bd8b-d0177976452f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-0a2577-7f97948b44-d669v\" (UID: \"7365fe23-d73c-467d-bd8b-d0177976452f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0a2577-7f97948b44-d669v" Oct 30 22:05:25.642041 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:25.642002 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0a2577-7f97948b44-d669v" Oct 30 22:05:25.937494 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:25.937462 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0a2577-7f97948b44-d669v"] Oct 30 22:05:25.938620 ip-10-0-137-181 kubenswrapper[6720]: W1030 22:05:25.938588 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7365fe23_d73c_467d_bd8b_d0177976452f.slice/crio-c4048ca82a867cfb8685bad7dacb13d9116a1f0ed6c96402c5ee3ff0d366c99b WatchSource:0}: Error finding container c4048ca82a867cfb8685bad7dacb13d9116a1f0ed6c96402c5ee3ff0d366c99b: Status 404 returned error can't find the container with id c4048ca82a867cfb8685bad7dacb13d9116a1f0ed6c96402c5ee3ff0d366c99b Oct 30 22:05:26.027978 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:26.027930 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:26.759457 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:26.759336 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0f73c9-88bff4c9f-qd4gf" Oct 30 22:05:26.795299 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:26.795260 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0a2577-7f97948b44-d669v" event={"ID":"7365fe23-d73c-467d-bd8b-d0177976452f","Type":"ContainerStarted","Data":"6aabc6ce1727c178c59ff358db85a05f4fb1bd5e7e7f669143faad61ccddee65"} Oct 30 22:05:26.795524 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:26.795325 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0a2577-7f97948b44-d669v" event={"ID":"7365fe23-d73c-467d-bd8b-d0177976452f","Type":"ContainerStarted","Data":"c4048ca82a867cfb8685bad7dacb13d9116a1f0ed6c96402c5ee3ff0d366c99b"} Oct 30 22:05:26.795731 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:26.795712 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0a2577-7f97948b44-d669v" Oct 30 22:05:26.860045 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:26.859981 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0a2577-7f97948b44-d669v" podStartSLOduration=3.420545403 podStartE2EDuration="3.85996368s" totalImagesPullingTime="439.418277ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:26.857892899 +0000 UTC m=+1493.028858108" watchObservedRunningTime="2026-10-30 22:05:26.85996368 +0000 UTC m=+1493.030928865" Oct 30 22:05:26.871630 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:26.871592 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:27.045087 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:27.045010 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:27.443210 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:27.443180 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:27.910736 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:27.910706 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:28.090695 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:28.090665 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:28.532182 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:28.529523 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:29.069043 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:29.069013 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:29.297777 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:29.297751 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:29.593200 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:29.593170 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:29.914233 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:29.914198 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:30.518793 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:30.518755 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:31.459409 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:31.453342 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:32.153751 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:32.153720 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:32.677086 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:32.677051 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:32.820498 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:32.819563 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0a2577-7f97948b44-d669v" Oct 30 22:05:33.368933 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:33.368902 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:33.686888 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:33.686799 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:34.289502 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:34.289469 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:35.041907 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:35.040249 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:35.861208 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:35.861164 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:36.137988 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:36.137906 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:36.794496 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:36.794460 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:37.907466 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:37.907435 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:38.720355 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:38.720315 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:39.527572 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:39.527537 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:40.227334 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:40.227279 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:40.403866 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:05:40.403827 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 22:05:40.638986 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:40.638953 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:40.881989 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:40.881771 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:41.387822 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:41.387787 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:41.630804 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:41.630585 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:42.709976 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:42.709940 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:43.074796 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:43.074707 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:43.508564 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:43.508523 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:43.946262 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:43.946228 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:44.232526 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:44.232403 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:45.014932 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:45.014894 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:45.738432 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:45.738258 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:46.415069 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:46.415041 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:47.126446 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:47.120240 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:47.370435 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:47.370397 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:47.837653 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:47.837592 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:47.979927 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:47.979858 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:48.138994 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:48.138947 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:48.854999 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:48.854966 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:49.639721 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:49.639577 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:50.314337 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:50.314287 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:50.927835 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:50.927800 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:51.733596 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:51.733560 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:52.433765 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:52.433734 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:53.227577 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:53.227534 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:53.907149 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:53.907078 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:54.059914 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:54.059877 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:54.741099 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:54.741069 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:55.361696 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:55.361648 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:56.176078 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:56.163305 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:56.836254 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:56.836194 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:57.534564 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:57.534532 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:58.208166 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:58.208132 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:58.932116 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:58.932080 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:59.088337 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:59.088291 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:05:59.708280 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:05:59.703880 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:00.449509 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:00.445407 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:01.146424 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:01.146347 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:01.826205 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:01.826159 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:02.428563 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:02.428515 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:03.141043 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:03.140983 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:03.802138 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:03.802063 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:04.440108 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:04.440065 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:05.421643 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:05.421603 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:06.558415 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:06.557882 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:07.419086 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:07.419050 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:08.116482 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:08.116434 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:08.322420 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:08.320527 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:08.473997 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:08.473907 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:08.739458 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:08.736934 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:09.084525 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:09.084435 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:10.012414 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:10.001945 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:10.412454 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:06:10.412411 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 22:06:10.625320 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:10.625287 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:11.274858 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:11.274809 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:12.006791 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:12.006747 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:12.593443 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:12.593344 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:12.852513 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:12.852416 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:13.533078 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:13.532779 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:14.118131 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:14.118093 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:14.332252 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:14.332206 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:15.131913 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:15.131865 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:15.853456 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:15.853419 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:16.520575 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:16.520527 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:17.251180 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:17.249727 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:18.210774 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:18.210742 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:18.953255 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:18.953212 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:19.836476 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:19.836432 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:20.609424 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:20.604588 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:20.751647 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:20.751618 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:21.149372 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:21.149340 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:21.305022 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:21.304988 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:21.964232 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:21.964200 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:22.733432 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:22.733361 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:23.419177 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:23.419129 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:24.089505 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:24.089470 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:24.367808 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:24.367768 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:24.873245 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:24.873181 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:25.759719 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:25.759679 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:26.432952 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:26.431858 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:27.138036 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:27.137955 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:27.835184 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:27.835151 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:28.148321 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:28.148287 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:28.520514 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:28.520066 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:29.143905 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:29.143531 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:29.843654 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:29.843617 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:30.529426 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:30.529257 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:31.231945 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:31.231914 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:31.825082 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:31.825044 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:32.545961 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:32.545930 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:33.180858 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:33.179374 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:33.837417 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:33.836539 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:34.549080 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:34.549043 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:35.212802 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:35.212767 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:35.933468 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:35.933429 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:36.579283 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:36.579234 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:37.344927 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:37.337841 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:38.133427 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:38.129204 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:38.736103 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:38.736066 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:39.523836 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:39.523803 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:40.368598 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:40.368563 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:40.457334 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:06:40.457284 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 22:06:41.150056 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:41.150021 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:41.470888 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:41.470796 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:41.625156 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:41.625117 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:42.047029 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:42.046992 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:43.078931 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:43.078890 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:43.835411 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:43.831524 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:44.275152 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:44.275120 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:44.424428 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:44.424371 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:44.566689 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:44.566592 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:45.123962 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:45.123927 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:45.879218 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:45.879184 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:46.482560 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:46.482524 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:46.858963 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:46.858928 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:47.011190 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:47.011160 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:47.219892 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:47.219800 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:47.908788 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:47.908758 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:48.545129 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:48.545085 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:49.220596 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:49.220560 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:49.921786 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:49.921756 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:50.778799 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:50.778752 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:51.129615 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:51.129574 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:51.491276 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:51.491183 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:52.240216 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:52.240183 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:52.403498 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:52.403462 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:52.921808 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:52.921664 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:53.609336 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:53.609299 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:54.419444 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:54.418968 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:55.161010 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:55.160975 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:55.847547 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:55.847510 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:56.587736 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:56.581625 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:57.340129 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:57.340086 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:58.048090 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:58.048059 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:58.777092 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:58.777059 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:58.929459 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:58.929416 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:59.200408 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:59.198911 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:06:59.518304 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:06:59.518207 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:00.613912 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:00.613556 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:01.416444 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:01.413188 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:02.039771 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:02.039674 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:02.660069 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:02.659782 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:03.327701 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:03.325947 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:03.939409 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:03.935997 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:04.641306 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:04.641229 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:05.419236 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:05.419202 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:06.030225 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:06.030031 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:06.685886 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:06.685846 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:07.335349 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:07.335311 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:08.036834 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:08.036792 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:08.710501 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:08.709519 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:09.369864 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:09.369829 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:09.927575 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:09.927540 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:10.087858 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:10.087824 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:10.404250 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:07:10.404205 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 22:07:10.622217 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:10.622184 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:11.407932 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:11.407855 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:12.029831 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:12.029748 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:12.892635 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:12.892602 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:13.825273 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:13.825216 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:14.635304 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:14.635267 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:15.182319 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:15.182283 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:15.374410 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:15.369005 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:15.520097 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:15.520016 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:15.778510 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:15.778433 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:15.974234 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:15.974204 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:16.563590 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:16.563553 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:17.296115 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:17.296031 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:18.025298 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:18.025261 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:18.638298 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:18.638260 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:19.244670 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:19.244619 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:20.008482 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:20.008448 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:20.613077 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:20.612986 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:21.300068 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:21.300035 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:22.040170 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:22.040132 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:22.481711 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:22.481666 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:22.634033 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:22.633978 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:22.817710 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:22.817616 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:22.962832 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:22.962784 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:23.513244 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:23.513207 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:24.310407 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:24.308437 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:25.129156 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:25.129122 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:25.956205 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:25.956172 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:26.726130 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:26.726092 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:27.433073 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:27.427444 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:28.134634 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:28.134602 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:28.324797 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:28.324769 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:28.763085 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:28.763054 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:29.415555 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:29.415516 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:30.308911 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:30.308872 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:31.023565 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:31.023533 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:31.637963 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:31.637923 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:32.299242 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:32.299203 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:32.991979 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:32.991944 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:33.618120 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:33.618020 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:34.304625 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:34.303913 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:35.016045 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:35.016011 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:35.715440 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:35.715371 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:36.419112 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:36.419065 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:37.027132 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:37.025008 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:37.635870 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:37.635833 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:38.239345 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:38.239287 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:38.878400 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:38.878355 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:39.505108 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:39.505075 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:40.239005 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:40.238964 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:40.402402 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:07:40.402351 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 22:07:40.935964 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:40.935933 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:41.606934 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:41.606134 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:42.305739 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:42.305701 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:42.938485 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:42.938447 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:43.813987 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:43.813908 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:44.460047 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:44.453735 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:45.160855 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:45.160822 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:45.903293 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:45.901162 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:46.589739 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:46.589683 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:47.356746 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:47.356714 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:47.889311 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:47.889285 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:48.045123 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:48.045089 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:48.331174 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:48.331102 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:48.491803 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:48.491705 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:48.929277 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:48.927425 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:49.608695 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:49.608603 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:50.285864 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:50.285796 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:50.950508 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:50.950475 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:51.714958 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:51.714926 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:52.479348 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:52.479313 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:52.651587 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:52.651549 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:53.259017 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:53.258978 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:53.724031 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:53.723992 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:53.892209 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:53.892179 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:54.191357 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:54.191324 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:54.968042 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:54.968005 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:55.628375 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:55.628336 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:56.487798 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:56.487765 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:57.115446 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:57.115415 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:57.717211 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:57.717167 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:57.881195 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:57.881163 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jhkhv"] Oct 30 22:07:57.928294 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:57.928261 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jhkhv"] Oct 30 22:07:57.928508 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:57.928373 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jhkhv" Oct 30 22:07:58.045630 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:58.045557 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de3025de-693a-4527-a04e-c9dd6d3caf89-credential-socket\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jhkhv\" (UID: \"de3025de-693a-4527-a04e-c9dd6d3caf89\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jhkhv" Oct 30 22:07:58.045772 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:58.045686 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de3025de-693a-4527-a04e-c9dd6d3caf89-istio-envoy\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jhkhv\" (UID: \"de3025de-693a-4527-a04e-c9dd6d3caf89\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jhkhv" Oct 30 22:07:58.045772 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:58.045752 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de3025de-693a-4527-a04e-c9dd6d3caf89-workload-certs\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jhkhv\" (UID: \"de3025de-693a-4527-a04e-c9dd6d3caf89\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jhkhv" Oct 30 22:07:58.045906 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:58.045884 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de3025de-693a-4527-a04e-c9dd6d3caf89-workload-socket\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jhkhv\" (UID: \"de3025de-693a-4527-a04e-c9dd6d3caf89\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jhkhv" Oct 30 22:07:58.045971 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:58.045933 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de3025de-693a-4527-a04e-c9dd6d3caf89-istio-token\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jhkhv\" (UID: \"de3025de-693a-4527-a04e-c9dd6d3caf89\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jhkhv" Oct 30 22:07:58.046038 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:58.045994 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-585ng\" (UniqueName: \"kubernetes.io/projected/de3025de-693a-4527-a04e-c9dd6d3caf89-kube-api-access-585ng\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jhkhv\" (UID: \"de3025de-693a-4527-a04e-c9dd6d3caf89\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jhkhv" Oct 30 22:07:58.046038 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:58.046026 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de3025de-693a-4527-a04e-c9dd6d3caf89-istio-podinfo\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jhkhv\" (UID: \"de3025de-693a-4527-a04e-c9dd6d3caf89\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jhkhv" Oct 30 22:07:58.046164 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:58.046102 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de3025de-693a-4527-a04e-c9dd6d3caf89-istio-data\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jhkhv\" (UID: \"de3025de-693a-4527-a04e-c9dd6d3caf89\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jhkhv" Oct 30 22:07:58.046226 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:58.046193 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de3025de-693a-4527-a04e-c9dd6d3caf89-istiod-ca-cert\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jhkhv\" (UID: \"de3025de-693a-4527-a04e-c9dd6d3caf89\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jhkhv" Oct 30 22:07:58.147974 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:58.147923 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de3025de-693a-4527-a04e-c9dd6d3caf89-workload-socket\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jhkhv\" (UID: \"de3025de-693a-4527-a04e-c9dd6d3caf89\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jhkhv" Oct 30 22:07:58.148182 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:58.147996 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de3025de-693a-4527-a04e-c9dd6d3caf89-istio-token\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jhkhv\" (UID: \"de3025de-693a-4527-a04e-c9dd6d3caf89\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jhkhv" Oct 30 22:07:58.148182 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:58.148026 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-585ng\" (UniqueName: \"kubernetes.io/projected/de3025de-693a-4527-a04e-c9dd6d3caf89-kube-api-access-585ng\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jhkhv\" (UID: \"de3025de-693a-4527-a04e-c9dd6d3caf89\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jhkhv" Oct 30 22:07:58.148182 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:58.148062 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de3025de-693a-4527-a04e-c9dd6d3caf89-istio-podinfo\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jhkhv\" (UID: \"de3025de-693a-4527-a04e-c9dd6d3caf89\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jhkhv" Oct 30 22:07:58.148182 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:58.148114 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de3025de-693a-4527-a04e-c9dd6d3caf89-istio-data\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jhkhv\" (UID: \"de3025de-693a-4527-a04e-c9dd6d3caf89\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jhkhv" Oct 30 22:07:58.148182 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:58.148170 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de3025de-693a-4527-a04e-c9dd6d3caf89-istiod-ca-cert\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jhkhv\" (UID: \"de3025de-693a-4527-a04e-c9dd6d3caf89\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jhkhv" Oct 30 22:07:58.148498 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:58.148247 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de3025de-693a-4527-a04e-c9dd6d3caf89-credential-socket\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jhkhv\" (UID: \"de3025de-693a-4527-a04e-c9dd6d3caf89\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jhkhv" Oct 30 22:07:58.148498 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:58.148328 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de3025de-693a-4527-a04e-c9dd6d3caf89-istio-envoy\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jhkhv\" (UID: \"de3025de-693a-4527-a04e-c9dd6d3caf89\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jhkhv" Oct 30 22:07:58.148498 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:58.148414 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de3025de-693a-4527-a04e-c9dd6d3caf89-workload-certs\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jhkhv\" (UID: \"de3025de-693a-4527-a04e-c9dd6d3caf89\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jhkhv" Oct 30 22:07:58.148498 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:58.148452 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de3025de-693a-4527-a04e-c9dd6d3caf89-workload-socket\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jhkhv\" (UID: \"de3025de-693a-4527-a04e-c9dd6d3caf89\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jhkhv" Oct 30 22:07:58.148705 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:58.148565 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de3025de-693a-4527-a04e-c9dd6d3caf89-istio-data\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jhkhv\" (UID: \"de3025de-693a-4527-a04e-c9dd6d3caf89\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jhkhv" Oct 30 22:07:58.148705 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:58.148698 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de3025de-693a-4527-a04e-c9dd6d3caf89-credential-socket\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jhkhv\" (UID: \"de3025de-693a-4527-a04e-c9dd6d3caf89\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jhkhv" Oct 30 22:07:58.148901 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:58.148874 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de3025de-693a-4527-a04e-c9dd6d3caf89-workload-certs\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jhkhv\" (UID: \"de3025de-693a-4527-a04e-c9dd6d3caf89\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jhkhv" Oct 30 22:07:58.148901 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:58.148890 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de3025de-693a-4527-a04e-c9dd6d3caf89-istiod-ca-cert\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jhkhv\" (UID: \"de3025de-693a-4527-a04e-c9dd6d3caf89\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jhkhv" Oct 30 22:07:58.150703 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:58.150684 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de3025de-693a-4527-a04e-c9dd6d3caf89-istio-envoy\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jhkhv\" (UID: \"de3025de-693a-4527-a04e-c9dd6d3caf89\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jhkhv" Oct 30 22:07:58.150830 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:58.150812 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de3025de-693a-4527-a04e-c9dd6d3caf89-istio-podinfo\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jhkhv\" (UID: \"de3025de-693a-4527-a04e-c9dd6d3caf89\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jhkhv" Oct 30 22:07:58.160896 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:58.160868 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de3025de-693a-4527-a04e-c9dd6d3caf89-istio-token\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jhkhv\" (UID: \"de3025de-693a-4527-a04e-c9dd6d3caf89\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jhkhv" Oct 30 22:07:58.161007 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:58.160977 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-585ng\" (UniqueName: \"kubernetes.io/projected/de3025de-693a-4527-a04e-c9dd6d3caf89-kube-api-access-585ng\") pod \"e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jhkhv\" (UID: \"de3025de-693a-4527-a04e-c9dd6d3caf89\") " pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jhkhv" Oct 30 22:07:58.248076 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:58.248042 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jhkhv" Oct 30 22:07:58.436512 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:58.435701 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:58.567165 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:58.567137 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jhkhv"] Oct 30 22:07:58.568466 ip-10-0-137-181 kubenswrapper[6720]: W1030 22:07:58.568435 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde3025de_693a_4527_a04e_c9dd6d3caf89.slice/crio-c6b81f151d530d07d39e98300ebe602916af6ebf71f968b96ce260a7f29140ba WatchSource:0}: Error finding container c6b81f151d530d07d39e98300ebe602916af6ebf71f968b96ce260a7f29140ba: Status 404 returned error can't find the container with id c6b81f151d530d07d39e98300ebe602916af6ebf71f968b96ce260a7f29140ba Oct 30 22:07:58.571546 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:58.571506 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:07:58.571654 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:58.571590 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:07:58.571654 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:58.571632 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:07:59.088840 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:59.088807 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jhkhv" event={"ID":"de3025de-693a-4527-a04e-c9dd6d3caf89","Type":"ContainerStarted","Data":"f4af5bcb39a183d6906be3f935d9f48a5e1fbd5fcbe4cb06dbd2d2d78f3be5dc"} Oct 30 22:07:59.089263 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:59.088866 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jhkhv" event={"ID":"de3025de-693a-4527-a04e-c9dd6d3caf89","Type":"ContainerStarted","Data":"c6b81f151d530d07d39e98300ebe602916af6ebf71f968b96ce260a7f29140ba"} Oct 30 22:07:59.108359 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:59.108319 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:07:59.116903 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:59.116165 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jhkhv" podStartSLOduration=2.11614533 podStartE2EDuration="2.11614533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:59.114892628 +0000 UTC m=+1645.285857811" watchObservedRunningTime="2026-10-30 22:07:59.11614533 +0000 UTC m=+1645.287110516" Oct 30 22:07:59.249803 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:59.249768 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jhkhv" Oct 30 22:07:59.260098 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:59.259558 6720 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jhkhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 30 22:07:59.260098 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:59.259655 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jhkhv" podUID="de3025de-693a-4527-a04e-c9dd6d3caf89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 30 22:07:59.843027 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:07:59.842993 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:00.260409 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:00.259775 6720 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jhkhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 30 22:08:00.260409 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:00.259839 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jhkhv" podUID="de3025de-693a-4527-a04e-c9dd6d3caf89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 30 22:08:00.514894 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:00.514809 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:01.162532 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:01.162501 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:01.250870 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:01.250833 6720 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jhkhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 30 22:08:01.251930 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:01.251898 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jhkhv" podUID="de3025de-693a-4527-a04e-c9dd6d3caf89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 30 22:08:01.923408 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:01.922501 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-0f1f9c-5f84576495-jbb2v"] Oct 30 22:08:01.932151 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:01.927705 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f1f9c-5f84576495-jbb2v" Oct 30 22:08:01.932813 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:01.932787 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-0f1f9c-serving-cert\"" Oct 30 22:08:01.937049 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:01.937021 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-0f1f9c-5f84576495-jbb2v"] Oct 30 22:08:02.021779 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:02.021747 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:02.113884 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:02.113847 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f0f9dee-25e5-4a4b-94e8-a919f3ca7d0a-openshift-service-ca\") pod \"maas-api-e2e-explicit-0f1f9c-5f84576495-jbb2v\" (UID: \"8f0f9dee-25e5-4a4b-94e8-a919f3ca7d0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f1f9c-5f84576495-jbb2v" Oct 30 22:08:02.114058 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:02.114043 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f0f9dee-25e5-4a4b-94e8-a919f3ca7d0a-metrics-certs\") pod \"maas-api-e2e-explicit-0f1f9c-5f84576495-jbb2v\" (UID: \"8f0f9dee-25e5-4a4b-94e8-a919f3ca7d0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f1f9c-5f84576495-jbb2v" Oct 30 22:08:02.114105 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:02.114087 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtvdv\" (UniqueName: \"kubernetes.io/projected/8f0f9dee-25e5-4a4b-94e8-a919f3ca7d0a-kube-api-access-wtvdv\") pod \"maas-api-e2e-explicit-0f1f9c-5f84576495-jbb2v\" (UID: \"8f0f9dee-25e5-4a4b-94e8-a919f3ca7d0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f1f9c-5f84576495-jbb2v" Oct 30 22:08:02.114177 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:02.114160 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f0f9dee-25e5-4a4b-94e8-a919f3ca7d0a-maas-api-tls\") pod \"maas-api-e2e-explicit-0f1f9c-5f84576495-jbb2v\" (UID: \"8f0f9dee-25e5-4a4b-94e8-a919f3ca7d0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f1f9c-5f84576495-jbb2v" Oct 30 22:08:02.215542 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:02.215458 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f0f9dee-25e5-4a4b-94e8-a919f3ca7d0a-openshift-service-ca\") pod \"maas-api-e2e-explicit-0f1f9c-5f84576495-jbb2v\" (UID: \"8f0f9dee-25e5-4a4b-94e8-a919f3ca7d0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f1f9c-5f84576495-jbb2v" Oct 30 22:08:02.215718 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:02.215647 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f0f9dee-25e5-4a4b-94e8-a919f3ca7d0a-metrics-certs\") pod \"maas-api-e2e-explicit-0f1f9c-5f84576495-jbb2v\" (UID: \"8f0f9dee-25e5-4a4b-94e8-a919f3ca7d0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f1f9c-5f84576495-jbb2v" Oct 30 22:08:02.215718 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:02.215695 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtvdv\" (UniqueName: \"kubernetes.io/projected/8f0f9dee-25e5-4a4b-94e8-a919f3ca7d0a-kube-api-access-wtvdv\") pod \"maas-api-e2e-explicit-0f1f9c-5f84576495-jbb2v\" (UID: \"8f0f9dee-25e5-4a4b-94e8-a919f3ca7d0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f1f9c-5f84576495-jbb2v" Oct 30 22:08:02.215853 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:02.215764 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f0f9dee-25e5-4a4b-94e8-a919f3ca7d0a-maas-api-tls\") pod \"maas-api-e2e-explicit-0f1f9c-5f84576495-jbb2v\" (UID: \"8f0f9dee-25e5-4a4b-94e8-a919f3ca7d0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f1f9c-5f84576495-jbb2v" Oct 30 22:08:02.215951 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:08:02.215929 6720 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-0f1f9c-serving-cert: secret "maas-api-e2e-explicit-0f1f9c-serving-cert" not found Oct 30 22:08:02.216043 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:08:02.216029 6720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f0f9dee-25e5-4a4b-94e8-a919f3ca7d0a-maas-api-tls podName:8f0f9dee-25e5-4a4b-94e8-a919f3ca7d0a nodeName:}" failed. No retries permitted until 2026-10-30 22:08:02.715998894 +0000 UTC m=+1648.886964061 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8f0f9dee-25e5-4a4b-94e8-a919f3ca7d0a-maas-api-tls") pod "maas-api-e2e-explicit-0f1f9c-5f84576495-jbb2v" (UID: "8f0f9dee-25e5-4a4b-94e8-a919f3ca7d0a") : secret "maas-api-e2e-explicit-0f1f9c-serving-cert" not found Oct 30 22:08:02.216274 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:02.216250 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f0f9dee-25e5-4a4b-94e8-a919f3ca7d0a-openshift-service-ca\") pod \"maas-api-e2e-explicit-0f1f9c-5f84576495-jbb2v\" (UID: \"8f0f9dee-25e5-4a4b-94e8-a919f3ca7d0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f1f9c-5f84576495-jbb2v" Oct 30 22:08:02.218539 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:02.218487 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f0f9dee-25e5-4a4b-94e8-a919f3ca7d0a-metrics-certs\") pod \"maas-api-e2e-explicit-0f1f9c-5f84576495-jbb2v\" (UID: \"8f0f9dee-25e5-4a4b-94e8-a919f3ca7d0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f1f9c-5f84576495-jbb2v" Oct 30 22:08:02.229853 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:02.229809 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:02.230724 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:02.230703 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtvdv\" (UniqueName: \"kubernetes.io/projected/8f0f9dee-25e5-4a4b-94e8-a919f3ca7d0a-kube-api-access-wtvdv\") pod \"maas-api-e2e-explicit-0f1f9c-5f84576495-jbb2v\" (UID: \"8f0f9dee-25e5-4a4b-94e8-a919f3ca7d0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f1f9c-5f84576495-jbb2v" Oct 30 22:08:02.253698 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:02.249724 6720 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jhkhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 30 22:08:02.253698 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:02.249785 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jhkhv" podUID="de3025de-693a-4527-a04e-c9dd6d3caf89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 30 22:08:02.374494 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:02.374458 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:02.728815 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:02.728221 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f0f9dee-25e5-4a4b-94e8-a919f3ca7d0a-maas-api-tls\") pod \"maas-api-e2e-explicit-0f1f9c-5f84576495-jbb2v\" (UID: \"8f0f9dee-25e5-4a4b-94e8-a919f3ca7d0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f1f9c-5f84576495-jbb2v" Oct 30 22:08:02.732154 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:02.732131 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f0f9dee-25e5-4a4b-94e8-a919f3ca7d0a-maas-api-tls\") pod \"maas-api-e2e-explicit-0f1f9c-5f84576495-jbb2v\" (UID: \"8f0f9dee-25e5-4a4b-94e8-a919f3ca7d0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f1f9c-5f84576495-jbb2v" Oct 30 22:08:02.878782 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:02.878749 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f1f9c-5f84576495-jbb2v" Oct 30 22:08:03.037794 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:03.037464 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:03.176852 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:03.176823 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-0f1f9c-5f84576495-jbb2v"] Oct 30 22:08:03.265858 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:03.265820 6720 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jhkhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 30 22:08:03.266129 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:03.266100 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jhkhv" podUID="de3025de-693a-4527-a04e-c9dd6d3caf89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 30 22:08:03.774628 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:03.774593 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:04.184768 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:04.184728 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f1f9c-5f84576495-jbb2v" event={"ID":"8f0f9dee-25e5-4a4b-94e8-a919f3ca7d0a","Type":"ContainerStarted","Data":"1cf211fb970d0f16fb22b8e8a3842bd02037a48df8f7111c19996a3f872ad942"} Oct 30 22:08:04.252406 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:04.251771 6720 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jhkhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 30 22:08:04.252406 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:04.251832 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jhkhv" podUID="de3025de-693a-4527-a04e-c9dd6d3caf89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 30 22:08:04.869410 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:04.868263 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:05.194703 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:05.194612 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f1f9c-5f84576495-jbb2v" event={"ID":"8f0f9dee-25e5-4a4b-94e8-a919f3ca7d0a","Type":"ContainerStarted","Data":"c052540a0d96fe796bbc698c6a1516eebb7a1d0d5211909e891af1cab418dfbd"} Oct 30 22:08:05.195231 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:05.195202 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f1f9c-5f84576495-jbb2v" Oct 30 22:08:05.248873 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:05.248840 6720 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jhkhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" start-of-body= Oct 30 22:08:05.249326 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:05.249197 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jhkhv" podUID="de3025de-693a-4527-a04e-c9dd6d3caf89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.101:15021/healthz/ready\": dial tcp 10.134.0.101:15021: connect: connection refused" Oct 30 22:08:05.842532 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:05.842485 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:06.676436 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:06.676378 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:06.772692 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:06.772658 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jhkhv" Oct 30 22:08:06.773492 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:06.773470 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jhkhv" Oct 30 22:08:06.774123 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:06.774087 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jhkhv" Oct 30 22:08:06.801273 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:06.801223 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f1f9c-5f84576495-jbb2v" podStartSLOduration=5.129176906 podStartE2EDuration="5.801210192s" totalImagesPullingTime="672.033286ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:05.219953131 +0000 UTC m=+1651.390918314" watchObservedRunningTime="2026-10-30 22:08:06.801210192 +0000 UTC m=+1652.972175382" Oct 30 22:08:07.312994 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:07.312959 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:08.118190 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:08.118153 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:08.847342 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:08.847312 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:09.501351 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:09.501254 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:10.145605 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:10.145534 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:10.405159 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:08:10.405073 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 22:08:10.742980 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:10.742891 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:11.208009 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:11.207976 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f1f9c-5f84576495-jbb2v" Oct 30 22:08:11.551693 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:11.551592 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:12.221229 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:12.221186 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:12.896914 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:12.896879 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:13.521944 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:13.521907 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:14.336935 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:14.336895 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:15.023449 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:15.023356 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:15.653140 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:15.653103 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:16.423366 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:16.423252 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:17.105608 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:17.105574 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:17.771220 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:17.771182 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:17.957566 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:17.957527 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:18.637443 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:18.637365 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:19.359408 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:19.355221 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:20.131319 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:20.131286 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:20.400856 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:20.400765 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:21.012023 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:21.011972 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:21.693131 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:21.693097 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:22.340408 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:22.333963 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:23.229417 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:23.225819 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:23.971696 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:23.971666 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:24.628217 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:24.628183 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:25.352067 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:25.352029 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:26.137030 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:26.136996 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:26.817018 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:26.816912 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:27.486345 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:27.486302 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:28.226669 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:28.225303 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:28.382597 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:28.382566 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:28.941296 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:28.941263 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:29.620567 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:29.620441 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:30.314633 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:30.314598 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:31.151157 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:31.151109 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:31.503460 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:31.503352 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:32.268000 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:32.267969 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:32.939479 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:32.939426 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:33.632070 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:33.632015 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:34.357056 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:34.357005 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:35.050694 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:35.050655 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:35.918413 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:35.911460 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:36.721248 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:36.719563 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:37.733892 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:37.733862 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:38.543004 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:38.542960 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:39.241945 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:39.241910 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:40.020041 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:40.019987 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:40.405459 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:08:40.405406 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 22:08:40.722038 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:40.721926 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:41.335854 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:41.335813 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gcxhh"] Oct 30 22:08:41.336079 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:41.336052 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gcxhh" podUID="ac5c00c4-2343-47bd-805d-adc1bdf138f6" containerName="istio-proxy" containerID="cri-o://5a8d3f6dabaf4a4a5c0c0c354fdcfa292bfb0a3e326b44ed03289be899c08f0b" gracePeriod=30 Oct 30 22:08:41.494120 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:41.494075 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:41.906816 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:41.906777 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:42.047092 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:42.047042 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:42.195013 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:42.194919 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:42.820663 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:42.820613 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:43.460278 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:43.460249 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:44.089670 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:44.089627 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:44.745426 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:44.745037 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:45.444427 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:45.444355 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:46.119425 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:46.110770 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:46.553413 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:46.552713 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-0f1f9c-5f84576495-jbb2v"] Oct 30 22:08:46.553413 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:46.553003 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f1f9c-5f84576495-jbb2v" podUID="8f0f9dee-25e5-4a4b-94e8-a919f3ca7d0a" containerName="maas-api" containerID="cri-o://c052540a0d96fe796bbc698c6a1516eebb7a1d0d5211909e891af1cab418dfbd" gracePeriod=30 Oct 30 22:08:46.557820 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:46.557643 6720 generic.go:356] "Generic (PLEG): container finished" podID="ac5c00c4-2343-47bd-805d-adc1bdf138f6" containerID="5a8d3f6dabaf4a4a5c0c0c354fdcfa292bfb0a3e326b44ed03289be899c08f0b" exitCode=0 Oct 30 22:08:46.557820 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:46.557745 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gcxhh" event={"ID":"ac5c00c4-2343-47bd-805d-adc1bdf138f6","Type":"ContainerDied","Data":"5a8d3f6dabaf4a4a5c0c0c354fdcfa292bfb0a3e326b44ed03289be899c08f0b"} Oct 30 22:08:46.640757 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:46.640670 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:46.719513 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:46.719485 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gcxhh" Oct 30 22:08:46.830663 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:46.828715 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac5c00c4-2343-47bd-805d-adc1bdf138f6-kube-api-access-jnn75\" (UniqueName: \"kubernetes.io/projected/ac5c00c4-2343-47bd-805d-adc1bdf138f6-kube-api-access-jnn75\") pod \"ac5c00c4-2343-47bd-805d-adc1bdf138f6\" (UID: \"ac5c00c4-2343-47bd-805d-adc1bdf138f6\") " Oct 30 22:08:46.830663 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:46.828781 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ac5c00c4-2343-47bd-805d-adc1bdf138f6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac5c00c4-2343-47bd-805d-adc1bdf138f6-istio-podinfo\") pod \"ac5c00c4-2343-47bd-805d-adc1bdf138f6\" (UID: \"ac5c00c4-2343-47bd-805d-adc1bdf138f6\") " Oct 30 22:08:46.830663 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:46.828816 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac5c00c4-2343-47bd-805d-adc1bdf138f6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac5c00c4-2343-47bd-805d-adc1bdf138f6-workload-certs\") pod \"ac5c00c4-2343-47bd-805d-adc1bdf138f6\" (UID: \"ac5c00c4-2343-47bd-805d-adc1bdf138f6\") " Oct 30 22:08:46.830663 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:46.828852 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac5c00c4-2343-47bd-805d-adc1bdf138f6-istio-token\" (UniqueName: \"kubernetes.io/projected/ac5c00c4-2343-47bd-805d-adc1bdf138f6-istio-token\") pod \"ac5c00c4-2343-47bd-805d-adc1bdf138f6\" (UID: \"ac5c00c4-2343-47bd-805d-adc1bdf138f6\") " Oct 30 22:08:46.830663 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:46.828896 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac5c00c4-2343-47bd-805d-adc1bdf138f6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac5c00c4-2343-47bd-805d-adc1bdf138f6-istio-envoy\") pod \"ac5c00c4-2343-47bd-805d-adc1bdf138f6\" (UID: \"ac5c00c4-2343-47bd-805d-adc1bdf138f6\") " Oct 30 22:08:46.830663 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:46.828947 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac5c00c4-2343-47bd-805d-adc1bdf138f6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac5c00c4-2343-47bd-805d-adc1bdf138f6-istio-data\") pod \"ac5c00c4-2343-47bd-805d-adc1bdf138f6\" (UID: \"ac5c00c4-2343-47bd-805d-adc1bdf138f6\") " Oct 30 22:08:46.830663 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:46.829071 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac5c00c4-2343-47bd-805d-adc1bdf138f6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac5c00c4-2343-47bd-805d-adc1bdf138f6-credential-socket\") pod \"ac5c00c4-2343-47bd-805d-adc1bdf138f6\" (UID: \"ac5c00c4-2343-47bd-805d-adc1bdf138f6\") " Oct 30 22:08:46.830663 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:46.829255 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac5c00c4-2343-47bd-805d-adc1bdf138f6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac5c00c4-2343-47bd-805d-adc1bdf138f6-istiod-ca-cert\") pod \"ac5c00c4-2343-47bd-805d-adc1bdf138f6\" (UID: \"ac5c00c4-2343-47bd-805d-adc1bdf138f6\") " Oct 30 22:08:46.830663 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:46.829284 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac5c00c4-2343-47bd-805d-adc1bdf138f6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac5c00c4-2343-47bd-805d-adc1bdf138f6-workload-socket\") pod \"ac5c00c4-2343-47bd-805d-adc1bdf138f6\" (UID: \"ac5c00c4-2343-47bd-805d-adc1bdf138f6\") " Oct 30 22:08:46.830663 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:46.830440 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac5c00c4-2343-47bd-805d-adc1bdf138f6-workload-socket" pod "ac5c00c4-2343-47bd-805d-adc1bdf138f6" (UID: "ac5c00c4-2343-47bd-805d-adc1bdf138f6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:08:46.875458 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:46.854510 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac5c00c4-2343-47bd-805d-adc1bdf138f6-workload-certs" pod "ac5c00c4-2343-47bd-805d-adc1bdf138f6" (UID: "ac5c00c4-2343-47bd-805d-adc1bdf138f6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:08:46.875458 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:46.854793 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac5c00c4-2343-47bd-805d-adc1bdf138f6-istio-data" pod "ac5c00c4-2343-47bd-805d-adc1bdf138f6" (UID: "ac5c00c4-2343-47bd-805d-adc1bdf138f6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:08:46.875458 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:46.859025 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac5c00c4-2343-47bd-805d-adc1bdf138f6-credential-socket" pod "ac5c00c4-2343-47bd-805d-adc1bdf138f6" (UID: "ac5c00c4-2343-47bd-805d-adc1bdf138f6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:08:46.875458 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:46.859515 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:46.875458 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:46.859561 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac5c00c4-2343-47bd-805d-adc1bdf138f6-istiod-ca-cert" pod "ac5c00c4-2343-47bd-805d-adc1bdf138f6" (UID: "ac5c00c4-2343-47bd-805d-adc1bdf138f6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:08:46.889432 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:46.888906 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac5c00c4-2343-47bd-805d-adc1bdf138f6-istio-envoy" pod "ac5c00c4-2343-47bd-805d-adc1bdf138f6" (UID: "ac5c00c4-2343-47bd-805d-adc1bdf138f6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:08:46.950469 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:46.933080 6720 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac5c00c4-2343-47bd-805d-adc1bdf138f6-workload-certs\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:46.950469 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:46.933115 6720 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac5c00c4-2343-47bd-805d-adc1bdf138f6-istio-envoy\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:46.950469 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:46.933131 6720 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac5c00c4-2343-47bd-805d-adc1bdf138f6-istio-data\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:46.950469 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:46.933143 6720 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac5c00c4-2343-47bd-805d-adc1bdf138f6-credential-socket\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:46.950469 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:46.933159 6720 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac5c00c4-2343-47bd-805d-adc1bdf138f6-istiod-ca-cert\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:46.950469 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:46.933173 6720 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac5c00c4-2343-47bd-805d-adc1bdf138f6-workload-socket\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:47.009793 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:47.009723 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ac5c00c4-2343-47bd-805d-adc1bdf138f6-istio-podinfo" pod "ac5c00c4-2343-47bd-805d-adc1bdf138f6" (UID: "ac5c00c4-2343-47bd-805d-adc1bdf138f6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:08:47.010251 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:47.010196 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac5c00c4-2343-47bd-805d-adc1bdf138f6-istio-token" pod "ac5c00c4-2343-47bd-805d-adc1bdf138f6" (UID: "ac5c00c4-2343-47bd-805d-adc1bdf138f6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:08:47.038097 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:47.035620 6720 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac5c00c4-2343-47bd-805d-adc1bdf138f6-istio-podinfo\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:47.038097 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:47.035674 6720 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac5c00c4-2343-47bd-805d-adc1bdf138f6-istio-token\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:47.041169 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:47.040952 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:47.108669 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:47.108606 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac5c00c4-2343-47bd-805d-adc1bdf138f6-kube-api-access-jnn75" pod "ac5c00c4-2343-47bd-805d-adc1bdf138f6" (UID: "ac5c00c4-2343-47bd-805d-adc1bdf138f6"). InnerVolumeSpecName "kube-api-access-jnn75". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:08:47.148306 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:47.148267 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnn75\" (UniqueName: \"kubernetes.io/projected/ac5c00c4-2343-47bd-805d-adc1bdf138f6-kube-api-access-jnn75\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:47.275696 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:47.275667 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f1f9c-5f84576495-jbb2v" Oct 30 22:08:47.352526 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:47.352483 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f0f9dee-25e5-4a4b-94e8-a919f3ca7d0a-kube-api-access-wtvdv\" (UniqueName: \"kubernetes.io/projected/8f0f9dee-25e5-4a4b-94e8-a919f3ca7d0a-kube-api-access-wtvdv\") pod \"8f0f9dee-25e5-4a4b-94e8-a919f3ca7d0a\" (UID: \"8f0f9dee-25e5-4a4b-94e8-a919f3ca7d0a\") " Oct 30 22:08:47.352764 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:47.352743 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f0f9dee-25e5-4a4b-94e8-a919f3ca7d0a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f0f9dee-25e5-4a4b-94e8-a919f3ca7d0a-openshift-service-ca\") pod \"8f0f9dee-25e5-4a4b-94e8-a919f3ca7d0a\" (UID: \"8f0f9dee-25e5-4a4b-94e8-a919f3ca7d0a\") " Oct 30 22:08:47.352926 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:47.352908 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f0f9dee-25e5-4a4b-94e8-a919f3ca7d0a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f0f9dee-25e5-4a4b-94e8-a919f3ca7d0a-metrics-certs\") pod \"8f0f9dee-25e5-4a4b-94e8-a919f3ca7d0a\" (UID: \"8f0f9dee-25e5-4a4b-94e8-a919f3ca7d0a\") " Oct 30 22:08:47.352995 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:47.352975 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f0f9dee-25e5-4a4b-94e8-a919f3ca7d0a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f0f9dee-25e5-4a4b-94e8-a919f3ca7d0a-maas-api-tls\") pod \"8f0f9dee-25e5-4a4b-94e8-a919f3ca7d0a\" (UID: \"8f0f9dee-25e5-4a4b-94e8-a919f3ca7d0a\") " Oct 30 22:08:47.354143 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:47.354109 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f0f9dee-25e5-4a4b-94e8-a919f3ca7d0a-openshift-service-ca" pod "8f0f9dee-25e5-4a4b-94e8-a919f3ca7d0a" (UID: "8f0f9dee-25e5-4a4b-94e8-a919f3ca7d0a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:08:47.356920 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:47.356889 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f0f9dee-25e5-4a4b-94e8-a919f3ca7d0a-maas-api-tls" pod "8f0f9dee-25e5-4a4b-94e8-a919f3ca7d0a" (UID: "8f0f9dee-25e5-4a4b-94e8-a919f3ca7d0a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:08:47.358038 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:47.358011 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f0f9dee-25e5-4a4b-94e8-a919f3ca7d0a-kube-api-access-wtvdv" pod "8f0f9dee-25e5-4a4b-94e8-a919f3ca7d0a" (UID: "8f0f9dee-25e5-4a4b-94e8-a919f3ca7d0a"). InnerVolumeSpecName "kube-api-access-wtvdv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:08:47.359679 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:47.359646 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f0f9dee-25e5-4a4b-94e8-a919f3ca7d0a-metrics-certs" pod "8f0f9dee-25e5-4a4b-94e8-a919f3ca7d0a" (UID: "8f0f9dee-25e5-4a4b-94e8-a919f3ca7d0a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:08:47.407196 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:47.407152 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:47.473413 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:47.470550 6720 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f0f9dee-25e5-4a4b-94e8-a919f3ca7d0a-maas-api-tls\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:47.473413 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:47.470588 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtvdv\" (UniqueName: \"kubernetes.io/projected/8f0f9dee-25e5-4a4b-94e8-a919f3ca7d0a-kube-api-access-wtvdv\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:47.473413 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:47.470603 6720 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f0f9dee-25e5-4a4b-94e8-a919f3ca7d0a-openshift-service-ca\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:47.473413 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:47.470615 6720 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f0f9dee-25e5-4a4b-94e8-a919f3ca7d0a-metrics-certs\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:47.567030 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:47.566934 6720 generic.go:356] "Generic (PLEG): container finished" podID="8f0f9dee-25e5-4a4b-94e8-a919f3ca7d0a" containerID="c052540a0d96fe796bbc698c6a1516eebb7a1d0d5211909e891af1cab418dfbd" exitCode=0 Oct 30 22:08:47.567518 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:47.567135 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f1f9c-5f84576495-jbb2v" event={"ID":"8f0f9dee-25e5-4a4b-94e8-a919f3ca7d0a","Type":"ContainerDied","Data":"c052540a0d96fe796bbc698c6a1516eebb7a1d0d5211909e891af1cab418dfbd"} Oct 30 22:08:47.567518 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:47.567223 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f1f9c-5f84576495-jbb2v" event={"ID":"8f0f9dee-25e5-4a4b-94e8-a919f3ca7d0a","Type":"ContainerDied","Data":"1cf211fb970d0f16fb22b8e8a3842bd02037a48df8f7111c19996a3f872ad942"} Oct 30 22:08:47.567518 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:47.567253 6720 scope.go:122] "RemoveContainer" containerID="c052540a0d96fe796bbc698c6a1516eebb7a1d0d5211909e891af1cab418dfbd" Oct 30 22:08:47.567518 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:47.567437 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-0f1f9c-5f84576495-jbb2v" Oct 30 22:08:47.573753 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:47.573717 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gcxhh" event={"ID":"ac5c00c4-2343-47bd-805d-adc1bdf138f6","Type":"ContainerDied","Data":"d7395ee923c5a4a3cf731f602337a3e595677322d86e14aa3909bf68f9fed24f"} Oct 30 22:08:47.573897 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:47.573791 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gcxhh" Oct 30 22:08:47.592799 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:47.591549 6720 scope.go:122] "RemoveContainer" containerID="c052540a0d96fe796bbc698c6a1516eebb7a1d0d5211909e891af1cab418dfbd" Oct 30 22:08:47.592799 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:08:47.592240 6720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c052540a0d96fe796bbc698c6a1516eebb7a1d0d5211909e891af1cab418dfbd\": container with ID starting with c052540a0d96fe796bbc698c6a1516eebb7a1d0d5211909e891af1cab418dfbd not found: ID does not exist" containerID="c052540a0d96fe796bbc698c6a1516eebb7a1d0d5211909e891af1cab418dfbd" Oct 30 22:08:47.592799 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:47.592287 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c052540a0d96fe796bbc698c6a1516eebb7a1d0d5211909e891af1cab418dfbd"} err="failed to get container status \"c052540a0d96fe796bbc698c6a1516eebb7a1d0d5211909e891af1cab418dfbd\": rpc error: code = NotFound desc = could not find container \"c052540a0d96fe796bbc698c6a1516eebb7a1d0d5211909e891af1cab418dfbd\": container with ID starting with c052540a0d96fe796bbc698c6a1516eebb7a1d0d5211909e891af1cab418dfbd not found: ID does not exist" Oct 30 22:08:47.592799 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:47.592313 6720 scope.go:122] "RemoveContainer" containerID="5a8d3f6dabaf4a4a5c0c0c354fdcfa292bfb0a3e326b44ed03289be899c08f0b" Oct 30 22:08:47.598274 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:47.598244 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:47.616374 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:47.616333 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-0f1f9c-5f84576495-jbb2v"] Oct 30 22:08:47.619913 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:47.619884 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-0f1f9c-5f84576495-jbb2v"] Oct 30 22:08:47.638447 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:47.633922 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gcxhh"] Oct 30 22:08:47.638946 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:47.638919 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-gcxhh"] Oct 30 22:08:47.958655 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:47.958619 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f0f9dee-25e5-4a4b-94e8-a919f3ca7d0a" path="/var/lib/kubelet/pods/8f0f9dee-25e5-4a4b-94e8-a919f3ca7d0a/volumes" Oct 30 22:08:47.959164 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:47.959149 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac5c00c4-2343-47bd-805d-adc1bdf138f6" path="/var/lib/kubelet/pods/ac5c00c4-2343-47bd-805d-adc1bdf138f6/volumes" Oct 30 22:08:48.223539 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:48.223464 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:48.990520 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:48.990484 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:49.729121 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:49.729056 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:50.561115 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:50.561085 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:51.266172 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:51.266122 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:51.959154 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:51.959121 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:52.747409 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:52.747197 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:53.434688 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:53.434650 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:54.139999 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:54.139901 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:54.331408 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:54.327262 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:54.494636 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:54.494563 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:54.704236 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:54.703995 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:54.874046 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:54.874019 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:55.341513 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:55.341425 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:55.547506 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:55.547357 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:55.846430 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:55.846372 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:56.018071 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:56.018031 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:56.449417 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:56.449364 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:57.254010 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:57.253960 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:57.849455 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:57.849425 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:58.006343 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:58.006310 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:58.167285 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:58.167257 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:58.469208 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:58.469127 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:58.958208 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:58.958170 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:08:59.524945 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:08:59.524866 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:00.352610 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:00.352576 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:01.067776 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:01.067741 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:01.707424 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:01.707353 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:02.522052 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:02.522019 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:03.246208 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:03.246175 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:04.025634 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:04.025602 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:04.713227 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:04.713176 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:05.526690 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:05.526656 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:06.330983 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:06.328130 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:07.069491 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:07.069458 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:07.763106 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:07.762159 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:08.530205 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:08.530161 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:09.320403 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:09.320359 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:10.401900 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:10.401858 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:10.492783 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:09:10.492747 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 22:09:10.561128 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:10.561090 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:10.778532 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:10.778449 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:11.641215 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:11.641139 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:12.404917 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:12.404882 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:13.248003 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:13.247972 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:13.980966 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:13.980938 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:14.760104 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:14.760057 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:15.576122 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:15.576087 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:15.742128 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:15.742090 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:16.304068 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:16.304021 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:17.055412 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:17.051135 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:17.838899 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:17.838789 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:18.635308 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:18.635270 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:19.431034 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:19.430924 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:20.335437 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:20.335405 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:21.019519 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:21.019485 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:21.819881 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:21.819842 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:22.621179 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:22.621144 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:23.272935 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:23.272898 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:23.938621 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:23.938548 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:24.617414 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:24.617364 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:25.356422 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:25.356346 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:26.043016 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:26.042978 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:26.745128 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:26.745081 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:27.428242 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:27.428206 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:28.104395 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:28.104356 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:28.381224 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:28.381147 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:28.998468 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:28.998434 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:29.636142 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:29.636103 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:30.646315 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:30.643196 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:31.362276 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:31.362216 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:32.056910 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:32.056874 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:32.719420 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:32.715681 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:33.465190 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:33.464052 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:34.134282 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:34.134239 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:34.773619 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:34.773577 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:35.307371 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:35.307332 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:35.718011 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:35.717965 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:36.324313 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:36.324280 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:36.896691 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:36.896651 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:37.259994 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:37.259923 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:37.944102 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:37.944045 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:38.619078 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:38.619029 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:39.364163 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:39.364129 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:40.035112 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:40.035082 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:40.403506 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:09:40.403467 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 22:09:40.830140 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:40.830025 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:41.619630 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:41.619591 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:42.311117 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:42.311059 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:43.090553 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:43.089794 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:43.856832 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:43.856663 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:44.708104 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:44.708069 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:45.494038 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:45.494003 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:46.165359 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:46.165322 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:46.809879 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:46.809840 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:47.504772 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:47.504702 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:48.134891 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:48.134845 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:48.778638 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:48.777905 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:48.923219 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:48.923189 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:49.371190 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:49.371158 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:49.543227 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:49.540085 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:49.878440 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:49.878407 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:50.530099 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:50.530055 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:51.206336 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:51.206301 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:51.909545 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:51.909507 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:52.826981 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:52.826941 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:53.459095 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:53.459067 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:53.619643 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:53.619608 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:54.320731 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:54.320688 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:54.691653 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:54.691620 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:54.891435 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:54.875981 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:55.230282 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:55.230220 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:56.228053 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:56.228010 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:57.032506 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:57.032467 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:57.737236 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:57.737187 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:58.429526 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:58.429491 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:58.575885 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:58.575850 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:59.009499 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:59.009464 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:59.156457 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:59.156428 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:59.332368 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:59.332278 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:09:59.919518 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:09:59.919481 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:00.642706 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:00.642666 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:01.479795 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:01.479763 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:02.340415 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:02.338581 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:02.703026 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:02.702989 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:02.968619 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:02.968546 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:03.436875 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:03.436848 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:03.931689 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:03.931652 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:04.123877 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:04.123831 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:04.162361 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:04.162336 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a39158-65fb5bdfd4-fpwzx"] Oct 30 22:10:04.163991 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:04.163963 6720 cpu_manager.go:414] "containerMap: removing container" podUID="ac5c00c4-2343-47bd-805d-adc1bdf138f6" containerName="istio-proxy" Oct 30 22:10:04.163991 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:04.163988 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac5c00c4-2343-47bd-805d-adc1bdf138f6" containerName="istio-proxy" Oct 30 22:10:04.164191 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:04.164013 6720 cpu_manager.go:414] "containerMap: removing container" podUID="8f0f9dee-25e5-4a4b-94e8-a919f3ca7d0a" containerName="maas-api" Oct 30 22:10:04.164191 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:04.164020 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f0f9dee-25e5-4a4b-94e8-a919f3ca7d0a" containerName="maas-api" Oct 30 22:10:04.164292 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:04.164209 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac5c00c4-2343-47bd-805d-adc1bdf138f6" containerName="istio-proxy" Oct 30 22:10:04.164292 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:04.164225 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f0f9dee-25e5-4a4b-94e8-a919f3ca7d0a" containerName="maas-api" Oct 30 22:10:04.175999 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:04.168058 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a39158-65fb5bdfd4-fpwzx" Oct 30 22:10:04.194964 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:04.194890 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a39158-65fb5bdfd4-fpwzx"] Oct 30 22:10:04.224445 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:04.224419 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-a39158-serving-cert\"" Oct 30 22:10:04.281996 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:04.281954 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e12f67b4-89de-4975-816d-f44fff4763f9-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-a39158-65fb5bdfd4-fpwzx\" (UID: \"e12f67b4-89de-4975-816d-f44fff4763f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a39158-65fb5bdfd4-fpwzx" Oct 30 22:10:04.282342 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:04.282317 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e12f67b4-89de-4975-816d-f44fff4763f9-metrics-certs\") pod \"maas-api-e2e-extmod-a-a39158-65fb5bdfd4-fpwzx\" (UID: \"e12f67b4-89de-4975-816d-f44fff4763f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a39158-65fb5bdfd4-fpwzx" Oct 30 22:10:04.282578 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:04.282552 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e12f67b4-89de-4975-816d-f44fff4763f9-maas-api-tls\") pod \"maas-api-e2e-extmod-a-a39158-65fb5bdfd4-fpwzx\" (UID: \"e12f67b4-89de-4975-816d-f44fff4763f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a39158-65fb5bdfd4-fpwzx" Oct 30 22:10:04.282681 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:04.282671 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msdcw\" (UniqueName: \"kubernetes.io/projected/e12f67b4-89de-4975-816d-f44fff4763f9-kube-api-access-msdcw\") pod \"maas-api-e2e-extmod-a-a39158-65fb5bdfd4-fpwzx\" (UID: \"e12f67b4-89de-4975-816d-f44fff4763f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a39158-65fb5bdfd4-fpwzx" Oct 30 22:10:04.283070 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:04.283049 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:04.383550 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:04.383509 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msdcw\" (UniqueName: \"kubernetes.io/projected/e12f67b4-89de-4975-816d-f44fff4763f9-kube-api-access-msdcw\") pod \"maas-api-e2e-extmod-a-a39158-65fb5bdfd4-fpwzx\" (UID: \"e12f67b4-89de-4975-816d-f44fff4763f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a39158-65fb5bdfd4-fpwzx" Oct 30 22:10:04.383753 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:04.383674 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e12f67b4-89de-4975-816d-f44fff4763f9-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-a39158-65fb5bdfd4-fpwzx\" (UID: \"e12f67b4-89de-4975-816d-f44fff4763f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a39158-65fb5bdfd4-fpwzx" Oct 30 22:10:04.383753 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:04.383697 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e12f67b4-89de-4975-816d-f44fff4763f9-metrics-certs\") pod \"maas-api-e2e-extmod-a-a39158-65fb5bdfd4-fpwzx\" (UID: \"e12f67b4-89de-4975-816d-f44fff4763f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a39158-65fb5bdfd4-fpwzx" Oct 30 22:10:04.383753 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:04.383744 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e12f67b4-89de-4975-816d-f44fff4763f9-maas-api-tls\") pod \"maas-api-e2e-extmod-a-a39158-65fb5bdfd4-fpwzx\" (UID: \"e12f67b4-89de-4975-816d-f44fff4763f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a39158-65fb5bdfd4-fpwzx" Oct 30 22:10:04.383922 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:10:04.383899 6720 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-a39158-serving-cert: secret "maas-api-e2e-extmod-a-a39158-serving-cert" not found Oct 30 22:10:04.383983 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:10:04.383976 6720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e12f67b4-89de-4975-816d-f44fff4763f9-maas-api-tls podName:e12f67b4-89de-4975-816d-f44fff4763f9 nodeName:}" failed. No retries permitted until 2026-10-30 22:10:04.883948858 +0000 UTC m=+1771.054914031 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e12f67b4-89de-4975-816d-f44fff4763f9-maas-api-tls") pod "maas-api-e2e-extmod-a-a39158-65fb5bdfd4-fpwzx" (UID: "e12f67b4-89de-4975-816d-f44fff4763f9") : secret "maas-api-e2e-extmod-a-a39158-serving-cert" not found Oct 30 22:10:04.384590 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:04.384442 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e12f67b4-89de-4975-816d-f44fff4763f9-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-a39158-65fb5bdfd4-fpwzx\" (UID: \"e12f67b4-89de-4975-816d-f44fff4763f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a39158-65fb5bdfd4-fpwzx" Oct 30 22:10:04.386648 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:04.386620 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e12f67b4-89de-4975-816d-f44fff4763f9-metrics-certs\") pod \"maas-api-e2e-extmod-a-a39158-65fb5bdfd4-fpwzx\" (UID: \"e12f67b4-89de-4975-816d-f44fff4763f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a39158-65fb5bdfd4-fpwzx" Oct 30 22:10:04.397577 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:04.397553 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msdcw\" (UniqueName: \"kubernetes.io/projected/e12f67b4-89de-4975-816d-f44fff4763f9-kube-api-access-msdcw\") pod \"maas-api-e2e-extmod-a-a39158-65fb5bdfd4-fpwzx\" (UID: \"e12f67b4-89de-4975-816d-f44fff4763f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a39158-65fb5bdfd4-fpwzx" Oct 30 22:10:04.678118 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:04.678083 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:04.896533 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:04.896501 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e12f67b4-89de-4975-816d-f44fff4763f9-maas-api-tls\") pod \"maas-api-e2e-extmod-a-a39158-65fb5bdfd4-fpwzx\" (UID: \"e12f67b4-89de-4975-816d-f44fff4763f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a39158-65fb5bdfd4-fpwzx" Oct 30 22:10:04.906461 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:04.904914 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e12f67b4-89de-4975-816d-f44fff4763f9-maas-api-tls\") pod \"maas-api-e2e-extmod-a-a39158-65fb5bdfd4-fpwzx\" (UID: \"e12f67b4-89de-4975-816d-f44fff4763f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a39158-65fb5bdfd4-fpwzx" Oct 30 22:10:05.001836 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:05.001746 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:05.101096 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:05.101056 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a39158-65fb5bdfd4-fpwzx" Oct 30 22:10:05.172549 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:05.172512 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:05.505979 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:05.505825 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a39158-65fb5bdfd4-fpwzx"] Oct 30 22:10:05.509238 ip-10-0-137-181 kubenswrapper[6720]: W1030 22:10:05.509193 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode12f67b4_89de_4975_816d_f44fff4763f9.slice/crio-3f163065b7169e2c004fb22540abfb304e11c4488aae2dacfdb97d077aa8bbf1 WatchSource:0}: Error finding container 3f163065b7169e2c004fb22540abfb304e11c4488aae2dacfdb97d077aa8bbf1: Status 404 returned error can't find the container with id 3f163065b7169e2c004fb22540abfb304e11c4488aae2dacfdb97d077aa8bbf1 Oct 30 22:10:05.512015 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:05.511990 6720 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:10:06.023409 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:06.023036 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:06.248181 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:06.248081 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a39158-65fb5bdfd4-fpwzx" event={"ID":"e12f67b4-89de-4975-816d-f44fff4763f9","Type":"ContainerStarted","Data":"c7f4811ae2b1f6cd3919febbfe3adbb96f48b56fb12e132a00f67edbb9b5b864"} Oct 30 22:10:06.248181 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:06.248153 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a39158-65fb5bdfd4-fpwzx" Oct 30 22:10:06.248181 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:06.248176 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a39158-65fb5bdfd4-fpwzx" event={"ID":"e12f67b4-89de-4975-816d-f44fff4763f9","Type":"ContainerStarted","Data":"3f163065b7169e2c004fb22540abfb304e11c4488aae2dacfdb97d077aa8bbf1"} Oct 30 22:10:06.271541 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:06.271484 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a39158-65fb5bdfd4-fpwzx" podStartSLOduration=1.788940392 podStartE2EDuration="2.271469456s" totalImagesPullingTime="482.529064ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:06.268854442 +0000 UTC m=+1772.439819634" watchObservedRunningTime="2026-10-30 22:10:06.271469456 +0000 UTC m=+1772.442434638" Oct 30 22:10:06.755154 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:06.753994 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:07.501786 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:07.501737 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:08.125082 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:08.124839 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:08.828822 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:08.828788 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:09.584442 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:09.581982 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:10.223318 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:10.223257 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:10.403891 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:10:10.403836 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 22:10:10.902564 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:10.902526 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:11.615318 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:11.615283 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:12.261852 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:12.261818 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a39158-65fb5bdfd4-fpwzx" Oct 30 22:10:12.354448 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:12.343080 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:13.041954 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:13.041915 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:13.464593 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:13.464552 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:13.825130 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:13.824575 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-1a5bdf-7d9c8596d4-lvfnt"] Oct 30 22:10:13.831442 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:13.829912 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1a5bdf-7d9c8596d4-lvfnt" Oct 30 22:10:13.838478 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:13.838450 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-1a5bdf-serving-cert\"" Oct 30 22:10:13.841038 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:13.841013 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-1a5bdf-7d9c8596d4-lvfnt"] Oct 30 22:10:13.862603 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:13.862573 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5267ee0d-42b1-482f-aa8c-f1cba3b748ea-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-1a5bdf-7d9c8596d4-lvfnt\" (UID: \"5267ee0d-42b1-482f-aa8c-f1cba3b748ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1a5bdf-7d9c8596d4-lvfnt" Oct 30 22:10:13.862928 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:13.862902 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5267ee0d-42b1-482f-aa8c-f1cba3b748ea-metrics-certs\") pod \"maas-api-e2e-extmod-a-1a5bdf-7d9c8596d4-lvfnt\" (UID: \"5267ee0d-42b1-482f-aa8c-f1cba3b748ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1a5bdf-7d9c8596d4-lvfnt" Oct 30 22:10:13.863239 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:13.863204 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5267ee0d-42b1-482f-aa8c-f1cba3b748ea-maas-api-tls\") pod \"maas-api-e2e-extmod-a-1a5bdf-7d9c8596d4-lvfnt\" (UID: \"5267ee0d-42b1-482f-aa8c-f1cba3b748ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1a5bdf-7d9c8596d4-lvfnt" Oct 30 22:10:13.863507 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:13.863479 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxmpn\" (UniqueName: \"kubernetes.io/projected/5267ee0d-42b1-482f-aa8c-f1cba3b748ea-kube-api-access-xxmpn\") pod \"maas-api-e2e-extmod-a-1a5bdf-7d9c8596d4-lvfnt\" (UID: \"5267ee0d-42b1-482f-aa8c-f1cba3b748ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1a5bdf-7d9c8596d4-lvfnt" Oct 30 22:10:13.922031 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:13.921997 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:13.965324 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:13.965287 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5267ee0d-42b1-482f-aa8c-f1cba3b748ea-maas-api-tls\") pod \"maas-api-e2e-extmod-a-1a5bdf-7d9c8596d4-lvfnt\" (UID: \"5267ee0d-42b1-482f-aa8c-f1cba3b748ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1a5bdf-7d9c8596d4-lvfnt" Oct 30 22:10:13.965494 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:13.965363 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxmpn\" (UniqueName: \"kubernetes.io/projected/5267ee0d-42b1-482f-aa8c-f1cba3b748ea-kube-api-access-xxmpn\") pod \"maas-api-e2e-extmod-a-1a5bdf-7d9c8596d4-lvfnt\" (UID: \"5267ee0d-42b1-482f-aa8c-f1cba3b748ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1a5bdf-7d9c8596d4-lvfnt" Oct 30 22:10:13.965578 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:13.965557 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5267ee0d-42b1-482f-aa8c-f1cba3b748ea-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-1a5bdf-7d9c8596d4-lvfnt\" (UID: \"5267ee0d-42b1-482f-aa8c-f1cba3b748ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1a5bdf-7d9c8596d4-lvfnt" Oct 30 22:10:13.965646 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:13.965598 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5267ee0d-42b1-482f-aa8c-f1cba3b748ea-metrics-certs\") pod \"maas-api-e2e-extmod-a-1a5bdf-7d9c8596d4-lvfnt\" (UID: \"5267ee0d-42b1-482f-aa8c-f1cba3b748ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1a5bdf-7d9c8596d4-lvfnt" Oct 30 22:10:13.966185 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:13.966166 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5267ee0d-42b1-482f-aa8c-f1cba3b748ea-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-1a5bdf-7d9c8596d4-lvfnt\" (UID: \"5267ee0d-42b1-482f-aa8c-f1cba3b748ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1a5bdf-7d9c8596d4-lvfnt" Oct 30 22:10:13.967958 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:13.967939 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5267ee0d-42b1-482f-aa8c-f1cba3b748ea-metrics-certs\") pod \"maas-api-e2e-extmod-a-1a5bdf-7d9c8596d4-lvfnt\" (UID: \"5267ee0d-42b1-482f-aa8c-f1cba3b748ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1a5bdf-7d9c8596d4-lvfnt" Oct 30 22:10:13.968133 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:13.968111 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5267ee0d-42b1-482f-aa8c-f1cba3b748ea-maas-api-tls\") pod \"maas-api-e2e-extmod-a-1a5bdf-7d9c8596d4-lvfnt\" (UID: \"5267ee0d-42b1-482f-aa8c-f1cba3b748ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1a5bdf-7d9c8596d4-lvfnt" Oct 30 22:10:13.980455 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:13.980432 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxmpn\" (UniqueName: \"kubernetes.io/projected/5267ee0d-42b1-482f-aa8c-f1cba3b748ea-kube-api-access-xxmpn\") pod \"maas-api-e2e-extmod-a-1a5bdf-7d9c8596d4-lvfnt\" (UID: \"5267ee0d-42b1-482f-aa8c-f1cba3b748ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1a5bdf-7d9c8596d4-lvfnt" Oct 30 22:10:14.075997 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:14.075920 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:14.151446 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:14.151410 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1a5bdf-7d9c8596d4-lvfnt" Oct 30 22:10:14.466289 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:14.466237 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-1a5bdf-7d9c8596d4-lvfnt"] Oct 30 22:10:14.467353 ip-10-0-137-181 kubenswrapper[6720]: W1030 22:10:14.467322 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5267ee0d_42b1_482f_aa8c_f1cba3b748ea.slice/crio-86e56842c20109138a0ea6b68e7bd8f8fe3d4dabd2bf253c1bb5e6ee926f8560 WatchSource:0}: Error finding container 86e56842c20109138a0ea6b68e7bd8f8fe3d4dabd2bf253c1bb5e6ee926f8560: Status 404 returned error can't find the container with id 86e56842c20109138a0ea6b68e7bd8f8fe3d4dabd2bf253c1bb5e6ee926f8560 Oct 30 22:10:14.845911 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:14.845728 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:15.013635 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:15.013606 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:15.322182 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:15.321943 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1a5bdf-7d9c8596d4-lvfnt" event={"ID":"5267ee0d-42b1-482f-aa8c-f1cba3b748ea","Type":"ContainerStarted","Data":"49b2cf4fad8c8bb29da368f7fbbc3a83a4205b52c0b3740b2a1607b6e97139e4"} Oct 30 22:10:15.322182 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:15.322002 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1a5bdf-7d9c8596d4-lvfnt" event={"ID":"5267ee0d-42b1-482f-aa8c-f1cba3b748ea","Type":"ContainerStarted","Data":"86e56842c20109138a0ea6b68e7bd8f8fe3d4dabd2bf253c1bb5e6ee926f8560"} Oct 30 22:10:15.322182 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:15.322157 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1a5bdf-7d9c8596d4-lvfnt" Oct 30 22:10:15.353028 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:15.352929 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1a5bdf-7d9c8596d4-lvfnt" podStartSLOduration=1.878924339 podStartE2EDuration="2.35290873s" totalImagesPullingTime="473.984391ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:15.347440146 +0000 UTC m=+1781.518405354" watchObservedRunningTime="2026-10-30 22:10:15.35290873 +0000 UTC m=+1781.523873914" Oct 30 22:10:15.480906 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:15.479561 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:15.724947 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:15.724893 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:15.886135 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:15.886101 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:16.522105 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:16.522030 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:17.431314 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:17.431265 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:18.323408 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:18.323147 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:18.906812 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:18.906735 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:19.068243 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:19.068212 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:19.335812 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:19.335742 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:19.529464 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:19.529433 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:20.141795 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:20.141754 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:20.302006 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:20.301971 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:20.557325 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:20.557240 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:20.734263 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:20.734227 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:21.057835 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:21.057791 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:21.327464 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:21.327354 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:21.340549 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:21.340520 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1a5bdf-7d9c8596d4-lvfnt" Oct 30 22:10:22.165170 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:22.165139 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:22.918900 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:22.918866 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:23.330464 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:23.330097 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:23.879466 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:23.879420 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:24.726054 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:24.725967 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:25.188564 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:25.188522 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:25.348144 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:25.348091 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:25.558788 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:25.558690 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:26.376377 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:26.376348 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-7t2lk"] Oct 30 22:10:26.377105 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:26.377069 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-7t2lk" podUID="0902dba2-ba96-4382-86b8-f332dd81a061" containerName="istio-proxy" containerID="cri-o://9db27ae96858ba075fc3e0f44706e73e277d88de5a1acbb018f1b609dda51c7b" gracePeriod=30 Oct 30 22:10:26.382414 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:26.380962 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:26.677724 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:26.677639 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:26.857493 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:26.853595 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:27.532191 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:27.532137 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:28.307961 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:28.307921 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:28.693119 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:28.693076 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:28.861329 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:28.861296 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:29.529635 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:29.529580 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:30.230877 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:30.230842 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:30.474517 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:30.474482 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:31.340424 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:31.340214 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:31.509202 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:31.509168 6720 generic.go:356] "Generic (PLEG): container finished" podID="0902dba2-ba96-4382-86b8-f332dd81a061" containerID="9db27ae96858ba075fc3e0f44706e73e277d88de5a1acbb018f1b609dda51c7b" exitCode=0 Oct 30 22:10:31.509352 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:31.509249 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-7t2lk" event={"ID":"0902dba2-ba96-4382-86b8-f332dd81a061","Type":"ContainerDied","Data":"9db27ae96858ba075fc3e0f44706e73e277d88de5a1acbb018f1b609dda51c7b"} Oct 30 22:10:31.619991 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:31.619814 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:31.768462 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:31.766219 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:31.785512 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:31.785467 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-7t2lk" Oct 30 22:10:31.889914 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:31.889810 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0902dba2-ba96-4382-86b8-f332dd81a061-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0902dba2-ba96-4382-86b8-f332dd81a061-workload-socket\") pod \"0902dba2-ba96-4382-86b8-f332dd81a061\" (UID: \"0902dba2-ba96-4382-86b8-f332dd81a061\") " Oct 30 22:10:31.890096 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:31.889913 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0902dba2-ba96-4382-86b8-f332dd81a061-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0902dba2-ba96-4382-86b8-f332dd81a061-workload-certs\") pod \"0902dba2-ba96-4382-86b8-f332dd81a061\" (UID: \"0902dba2-ba96-4382-86b8-f332dd81a061\") " Oct 30 22:10:31.890096 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:31.889964 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0902dba2-ba96-4382-86b8-f332dd81a061-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0902dba2-ba96-4382-86b8-f332dd81a061-istio-envoy\") pod \"0902dba2-ba96-4382-86b8-f332dd81a061\" (UID: \"0902dba2-ba96-4382-86b8-f332dd81a061\") " Oct 30 22:10:31.890096 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:31.890005 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0902dba2-ba96-4382-86b8-f332dd81a061-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0902dba2-ba96-4382-86b8-f332dd81a061-credential-socket\") pod \"0902dba2-ba96-4382-86b8-f332dd81a061\" (UID: \"0902dba2-ba96-4382-86b8-f332dd81a061\") " Oct 30 22:10:31.890096 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:31.890042 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0902dba2-ba96-4382-86b8-f332dd81a061-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0902dba2-ba96-4382-86b8-f332dd81a061-istio-podinfo\") pod \"0902dba2-ba96-4382-86b8-f332dd81a061\" (UID: \"0902dba2-ba96-4382-86b8-f332dd81a061\") " Oct 30 22:10:31.890329 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:31.890156 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0902dba2-ba96-4382-86b8-f332dd81a061-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0902dba2-ba96-4382-86b8-f332dd81a061-istiod-ca-cert\") pod \"0902dba2-ba96-4382-86b8-f332dd81a061\" (UID: \"0902dba2-ba96-4382-86b8-f332dd81a061\") " Oct 30 22:10:31.890329 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:31.890199 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0902dba2-ba96-4382-86b8-f332dd81a061-kube-api-access-5rz9t\" (UniqueName: \"kubernetes.io/projected/0902dba2-ba96-4382-86b8-f332dd81a061-kube-api-access-5rz9t\") pod \"0902dba2-ba96-4382-86b8-f332dd81a061\" (UID: \"0902dba2-ba96-4382-86b8-f332dd81a061\") " Oct 30 22:10:31.890329 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:31.890275 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0902dba2-ba96-4382-86b8-f332dd81a061-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0902dba2-ba96-4382-86b8-f332dd81a061-istio-data\") pod \"0902dba2-ba96-4382-86b8-f332dd81a061\" (UID: \"0902dba2-ba96-4382-86b8-f332dd81a061\") " Oct 30 22:10:31.890519 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:31.890487 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0902dba2-ba96-4382-86b8-f332dd81a061-istio-token\" (UniqueName: \"kubernetes.io/projected/0902dba2-ba96-4382-86b8-f332dd81a061-istio-token\") pod \"0902dba2-ba96-4382-86b8-f332dd81a061\" (UID: \"0902dba2-ba96-4382-86b8-f332dd81a061\") " Oct 30 22:10:31.890734 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:31.890709 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0902dba2-ba96-4382-86b8-f332dd81a061-credential-socket" pod "0902dba2-ba96-4382-86b8-f332dd81a061" (UID: "0902dba2-ba96-4382-86b8-f332dd81a061"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:31.890894 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:31.890877 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0902dba2-ba96-4382-86b8-f332dd81a061-workload-socket" pod "0902dba2-ba96-4382-86b8-f332dd81a061" (UID: "0902dba2-ba96-4382-86b8-f332dd81a061"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:31.891007 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:31.890993 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0902dba2-ba96-4382-86b8-f332dd81a061-workload-certs" pod "0902dba2-ba96-4382-86b8-f332dd81a061" (UID: "0902dba2-ba96-4382-86b8-f332dd81a061"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:31.891137 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:31.891113 6720 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0902dba2-ba96-4382-86b8-f332dd81a061-workload-certs\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:31.891221 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:31.891145 6720 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0902dba2-ba96-4382-86b8-f332dd81a061-credential-socket\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:31.891221 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:31.891162 6720 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0902dba2-ba96-4382-86b8-f332dd81a061-workload-socket\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:31.891657 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:31.891631 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0902dba2-ba96-4382-86b8-f332dd81a061-istio-data" pod "0902dba2-ba96-4382-86b8-f332dd81a061" (UID: "0902dba2-ba96-4382-86b8-f332dd81a061"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:31.892315 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:31.892293 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0902dba2-ba96-4382-86b8-f332dd81a061-istiod-ca-cert" pod "0902dba2-ba96-4382-86b8-f332dd81a061" (UID: "0902dba2-ba96-4382-86b8-f332dd81a061"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:10:31.894065 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:31.894039 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0902dba2-ba96-4382-86b8-f332dd81a061-istio-podinfo" pod "0902dba2-ba96-4382-86b8-f332dd81a061" (UID: "0902dba2-ba96-4382-86b8-f332dd81a061"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:10:31.894255 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:31.894236 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0902dba2-ba96-4382-86b8-f332dd81a061-istio-envoy" pod "0902dba2-ba96-4382-86b8-f332dd81a061" (UID: "0902dba2-ba96-4382-86b8-f332dd81a061"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:31.895914 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:31.895877 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0902dba2-ba96-4382-86b8-f332dd81a061-istio-token" pod "0902dba2-ba96-4382-86b8-f332dd81a061" (UID: "0902dba2-ba96-4382-86b8-f332dd81a061"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:10:31.896123 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:31.896098 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0902dba2-ba96-4382-86b8-f332dd81a061-kube-api-access-5rz9t" pod "0902dba2-ba96-4382-86b8-f332dd81a061" (UID: "0902dba2-ba96-4382-86b8-f332dd81a061"). InnerVolumeSpecName "kube-api-access-5rz9t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:10:31.992450 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:31.992363 6720 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0902dba2-ba96-4382-86b8-f332dd81a061-istiod-ca-cert\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:31.992450 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:31.992452 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rz9t\" (UniqueName: \"kubernetes.io/projected/0902dba2-ba96-4382-86b8-f332dd81a061-kube-api-access-5rz9t\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:31.992687 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:31.992471 6720 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0902dba2-ba96-4382-86b8-f332dd81a061-istio-data\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:31.992687 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:31.992488 6720 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0902dba2-ba96-4382-86b8-f332dd81a061-istio-token\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:31.992687 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:31.992502 6720 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0902dba2-ba96-4382-86b8-f332dd81a061-istio-envoy\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:31.992687 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:31.992516 6720 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0902dba2-ba96-4382-86b8-f332dd81a061-istio-podinfo\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:32.482412 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:32.472302 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:32.521183 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:32.521157 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-7t2lk" Oct 30 22:10:32.521183 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:32.521180 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-7t2lk" event={"ID":"0902dba2-ba96-4382-86b8-f332dd81a061","Type":"ContainerDied","Data":"17811af6a3b46b669481c6862e1c3add3700ee146fbbb402717db3ae5972c2b1"} Oct 30 22:10:32.521447 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:32.521245 6720 scope.go:122] "RemoveContainer" containerID="9db27ae96858ba075fc3e0f44706e73e277d88de5a1acbb018f1b609dda51c7b" Oct 30 22:10:32.566025 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:32.557857 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-7t2lk"] Oct 30 22:10:32.566025 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:32.560246 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-7t2lk"] Oct 30 22:10:33.040964 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:33.040932 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-bacea0-58688dc49f-m4cv7"] Oct 30 22:10:33.041700 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:33.041673 6720 cpu_manager.go:414] "containerMap: removing container" podUID="0902dba2-ba96-4382-86b8-f332dd81a061" containerName="istio-proxy" Oct 30 22:10:33.041700 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:33.041691 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0902dba2-ba96-4382-86b8-f332dd81a061" containerName="istio-proxy" Oct 30 22:10:33.041903 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:33.041788 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="0902dba2-ba96-4382-86b8-f332dd81a061" containerName="istio-proxy" Oct 30 22:10:33.046927 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:33.046902 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bacea0-58688dc49f-m4cv7" Oct 30 22:10:33.052247 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:33.052220 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-bacea0-serving-cert\"" Oct 30 22:10:33.056651 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:33.056407 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-bacea0-58688dc49f-m4cv7"] Oct 30 22:10:33.113375 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:33.113342 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/132ce48a-6190-4f71-8b77-1a28c917030d-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-bacea0-58688dc49f-m4cv7\" (UID: \"132ce48a-6190-4f71-8b77-1a28c917030d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bacea0-58688dc49f-m4cv7" Oct 30 22:10:33.113624 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:33.113454 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnk5k\" (UniqueName: \"kubernetes.io/projected/132ce48a-6190-4f71-8b77-1a28c917030d-kube-api-access-xnk5k\") pod \"maas-api-e2e-extmod-a-bacea0-58688dc49f-m4cv7\" (UID: \"132ce48a-6190-4f71-8b77-1a28c917030d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bacea0-58688dc49f-m4cv7" Oct 30 22:10:33.113624 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:33.113607 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/132ce48a-6190-4f71-8b77-1a28c917030d-maas-api-tls\") pod \"maas-api-e2e-extmod-a-bacea0-58688dc49f-m4cv7\" (UID: \"132ce48a-6190-4f71-8b77-1a28c917030d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bacea0-58688dc49f-m4cv7" Oct 30 22:10:33.113765 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:33.113632 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/132ce48a-6190-4f71-8b77-1a28c917030d-metrics-certs\") pod \"maas-api-e2e-extmod-a-bacea0-58688dc49f-m4cv7\" (UID: \"132ce48a-6190-4f71-8b77-1a28c917030d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bacea0-58688dc49f-m4cv7" Oct 30 22:10:33.113765 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:33.113740 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:33.214358 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:33.214318 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/132ce48a-6190-4f71-8b77-1a28c917030d-maas-api-tls\") pod \"maas-api-e2e-extmod-a-bacea0-58688dc49f-m4cv7\" (UID: \"132ce48a-6190-4f71-8b77-1a28c917030d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bacea0-58688dc49f-m4cv7" Oct 30 22:10:33.214358 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:33.214357 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/132ce48a-6190-4f71-8b77-1a28c917030d-metrics-certs\") pod \"maas-api-e2e-extmod-a-bacea0-58688dc49f-m4cv7\" (UID: \"132ce48a-6190-4f71-8b77-1a28c917030d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bacea0-58688dc49f-m4cv7" Oct 30 22:10:33.214663 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:33.214409 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/132ce48a-6190-4f71-8b77-1a28c917030d-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-bacea0-58688dc49f-m4cv7\" (UID: \"132ce48a-6190-4f71-8b77-1a28c917030d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bacea0-58688dc49f-m4cv7" Oct 30 22:10:33.214663 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:33.214468 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnk5k\" (UniqueName: \"kubernetes.io/projected/132ce48a-6190-4f71-8b77-1a28c917030d-kube-api-access-xnk5k\") pod \"maas-api-e2e-extmod-a-bacea0-58688dc49f-m4cv7\" (UID: \"132ce48a-6190-4f71-8b77-1a28c917030d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bacea0-58688dc49f-m4cv7" Oct 30 22:10:33.215166 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:33.215103 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/132ce48a-6190-4f71-8b77-1a28c917030d-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-bacea0-58688dc49f-m4cv7\" (UID: \"132ce48a-6190-4f71-8b77-1a28c917030d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bacea0-58688dc49f-m4cv7" Oct 30 22:10:33.217131 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:33.217107 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/132ce48a-6190-4f71-8b77-1a28c917030d-maas-api-tls\") pod \"maas-api-e2e-extmod-a-bacea0-58688dc49f-m4cv7\" (UID: \"132ce48a-6190-4f71-8b77-1a28c917030d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bacea0-58688dc49f-m4cv7" Oct 30 22:10:33.217213 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:33.217107 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/132ce48a-6190-4f71-8b77-1a28c917030d-metrics-certs\") pod \"maas-api-e2e-extmod-a-bacea0-58688dc49f-m4cv7\" (UID: \"132ce48a-6190-4f71-8b77-1a28c917030d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bacea0-58688dc49f-m4cv7" Oct 30 22:10:33.230902 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:33.230880 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnk5k\" (UniqueName: \"kubernetes.io/projected/132ce48a-6190-4f71-8b77-1a28c917030d-kube-api-access-xnk5k\") pod \"maas-api-e2e-extmod-a-bacea0-58688dc49f-m4cv7\" (UID: \"132ce48a-6190-4f71-8b77-1a28c917030d\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bacea0-58688dc49f-m4cv7" Oct 30 22:10:33.267999 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:33.267961 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:33.364505 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:33.364471 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bacea0-58688dc49f-m4cv7" Oct 30 22:10:33.707947 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:33.707916 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-bacea0-58688dc49f-m4cv7"] Oct 30 22:10:33.958205 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:33.958125 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0902dba2-ba96-4382-86b8-f332dd81a061" path="/var/lib/kubelet/pods/0902dba2-ba96-4382-86b8-f332dd81a061/volumes" Oct 30 22:10:34.082957 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:34.082871 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:34.497138 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:34.497055 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:34.540901 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:34.540867 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bacea0-58688dc49f-m4cv7" event={"ID":"132ce48a-6190-4f71-8b77-1a28c917030d","Type":"ContainerStarted","Data":"e23c19265f41345552b83a4b4160b4d096b58b4e126e3777ca42e8f6497dbfee"} Oct 30 22:10:34.541088 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:34.540924 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bacea0-58688dc49f-m4cv7" event={"ID":"132ce48a-6190-4f71-8b77-1a28c917030d","Type":"ContainerStarted","Data":"af71e4ba7de3011808a6e0423695dc00d10d84eaa2810dcf4a330acc8dc2e164"} Oct 30 22:10:34.541088 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:34.541004 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bacea0-58688dc49f-m4cv7" Oct 30 22:10:34.564672 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:34.564622 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bacea0-58688dc49f-m4cv7" podStartSLOduration=1.124689373 podStartE2EDuration="1.564606913s" totalImagesPullingTime="439.91754ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:34.563624926 +0000 UTC m=+1800.734590121" watchObservedRunningTime="2026-10-30 22:10:34.564606913 +0000 UTC m=+1800.735572094" Oct 30 22:10:34.940727 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:34.940686 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:35.094409 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:35.093754 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:35.492464 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:35.492436 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:35.647076 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:35.647043 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:36.322126 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:36.321851 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:36.706654 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:36.706620 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:36.902413 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:36.881911 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:37.332419 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:37.330956 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:38.146984 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:38.146943 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:38.976820 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:38.969770 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:39.827005 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:39.826943 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:40.010981 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:40.010943 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:40.403636 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:10:40.403597 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 22:10:40.575016 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:40.574986 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bacea0-58688dc49f-m4cv7" Oct 30 22:10:40.696087 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:40.695998 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-26a128-846fd4d8c6-z4mcb"] Oct 30 22:10:40.703761 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:40.703724 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-26a128-846fd4d8c6-z4mcb" Oct 30 22:10:40.718183 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:40.718155 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-26a128-serving-cert\"" Oct 30 22:10:40.722340 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:40.722313 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-26a128-846fd4d8c6-z4mcb"] Oct 30 22:10:40.724156 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:40.724132 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c5ce4b5-7597-47c4-9530-56544816a191-metrics-certs\") pod \"maas-api-e2e-extmod-b-26a128-846fd4d8c6-z4mcb\" (UID: \"9c5ce4b5-7597-47c4-9530-56544816a191\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-26a128-846fd4d8c6-z4mcb" Oct 30 22:10:40.724417 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:40.724357 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c5ce4b5-7597-47c4-9530-56544816a191-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-26a128-846fd4d8c6-z4mcb\" (UID: \"9c5ce4b5-7597-47c4-9530-56544816a191\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-26a128-846fd4d8c6-z4mcb" Oct 30 22:10:40.724560 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:40.724534 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c5ce4b5-7597-47c4-9530-56544816a191-maas-api-tls\") pod \"maas-api-e2e-extmod-b-26a128-846fd4d8c6-z4mcb\" (UID: \"9c5ce4b5-7597-47c4-9530-56544816a191\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-26a128-846fd4d8c6-z4mcb" Oct 30 22:10:40.724667 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:40.724617 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwlpx\" (UniqueName: \"kubernetes.io/projected/9c5ce4b5-7597-47c4-9530-56544816a191-kube-api-access-bwlpx\") pod \"maas-api-e2e-extmod-b-26a128-846fd4d8c6-z4mcb\" (UID: \"9c5ce4b5-7597-47c4-9530-56544816a191\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-26a128-846fd4d8c6-z4mcb" Oct 30 22:10:40.826545 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:40.826488 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c5ce4b5-7597-47c4-9530-56544816a191-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-26a128-846fd4d8c6-z4mcb\" (UID: \"9c5ce4b5-7597-47c4-9530-56544816a191\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-26a128-846fd4d8c6-z4mcb" Oct 30 22:10:40.826750 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:40.826680 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c5ce4b5-7597-47c4-9530-56544816a191-maas-api-tls\") pod \"maas-api-e2e-extmod-b-26a128-846fd4d8c6-z4mcb\" (UID: \"9c5ce4b5-7597-47c4-9530-56544816a191\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-26a128-846fd4d8c6-z4mcb" Oct 30 22:10:40.826826 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:40.826747 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwlpx\" (UniqueName: \"kubernetes.io/projected/9c5ce4b5-7597-47c4-9530-56544816a191-kube-api-access-bwlpx\") pod \"maas-api-e2e-extmod-b-26a128-846fd4d8c6-z4mcb\" (UID: \"9c5ce4b5-7597-47c4-9530-56544816a191\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-26a128-846fd4d8c6-z4mcb" Oct 30 22:10:40.826826 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:40.826784 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c5ce4b5-7597-47c4-9530-56544816a191-metrics-certs\") pod \"maas-api-e2e-extmod-b-26a128-846fd4d8c6-z4mcb\" (UID: \"9c5ce4b5-7597-47c4-9530-56544816a191\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-26a128-846fd4d8c6-z4mcb" Oct 30 22:10:40.827259 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:40.827236 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c5ce4b5-7597-47c4-9530-56544816a191-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-26a128-846fd4d8c6-z4mcb\" (UID: \"9c5ce4b5-7597-47c4-9530-56544816a191\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-26a128-846fd4d8c6-z4mcb" Oct 30 22:10:40.829496 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:40.829476 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c5ce4b5-7597-47c4-9530-56544816a191-maas-api-tls\") pod \"maas-api-e2e-extmod-b-26a128-846fd4d8c6-z4mcb\" (UID: \"9c5ce4b5-7597-47c4-9530-56544816a191\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-26a128-846fd4d8c6-z4mcb" Oct 30 22:10:40.829595 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:40.829570 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c5ce4b5-7597-47c4-9530-56544816a191-metrics-certs\") pod \"maas-api-e2e-extmod-b-26a128-846fd4d8c6-z4mcb\" (UID: \"9c5ce4b5-7597-47c4-9530-56544816a191\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-26a128-846fd4d8c6-z4mcb" Oct 30 22:10:40.832854 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:40.832833 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:40.840915 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:40.840888 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwlpx\" (UniqueName: \"kubernetes.io/projected/9c5ce4b5-7597-47c4-9530-56544816a191-kube-api-access-bwlpx\") pod \"maas-api-e2e-extmod-b-26a128-846fd4d8c6-z4mcb\" (UID: \"9c5ce4b5-7597-47c4-9530-56544816a191\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-26a128-846fd4d8c6-z4mcb" Oct 30 22:10:41.008319 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:41.008240 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:41.020258 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:41.020231 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-26a128-846fd4d8c6-z4mcb" Oct 30 22:10:41.284792 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:41.284708 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:41.313592 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:41.313554 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-26a128-846fd4d8c6-z4mcb"] Oct 30 22:10:41.324336 ip-10-0-137-181 kubenswrapper[6720]: W1030 22:10:41.324277 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c5ce4b5_7597_47c4_9530_56544816a191.slice/crio-e9a2972280b52ff309ff17490d665131e1b0e40c79edbda69294a34d967568af WatchSource:0}: Error finding container e9a2972280b52ff309ff17490d665131e1b0e40c79edbda69294a34d967568af: Status 404 returned error can't find the container with id e9a2972280b52ff309ff17490d665131e1b0e40c79edbda69294a34d967568af Oct 30 22:10:41.602545 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:41.588765 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:41.656452 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:41.654368 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-26a128-846fd4d8c6-z4mcb" event={"ID":"9c5ce4b5-7597-47c4-9530-56544816a191","Type":"ContainerStarted","Data":"e9a2972280b52ff309ff17490d665131e1b0e40c79edbda69294a34d967568af"} Oct 30 22:10:41.923037 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:41.918957 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:42.662633 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:42.662597 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-26a128-846fd4d8c6-z4mcb" event={"ID":"9c5ce4b5-7597-47c4-9530-56544816a191","Type":"ContainerStarted","Data":"b2cdbbbd6e47e3342bc72ff84d140aca08fe5e40e3cffc12e4da7635c07c50cf"} Oct 30 22:10:42.663066 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:42.662658 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-26a128-846fd4d8c6-z4mcb" Oct 30 22:10:42.686985 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:42.686932 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-26a128-846fd4d8c6-z4mcb" podStartSLOduration=2.288679258 podStartE2EDuration="2.68691846s" totalImagesPullingTime="398.239202ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:42.683645116 +0000 UTC m=+1808.854610298" watchObservedRunningTime="2026-10-30 22:10:42.68691846 +0000 UTC m=+1808.857883642" Oct 30 22:10:42.728988 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:42.728952 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:43.415004 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:43.414928 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:43.575211 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:43.575177 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:44.054880 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:44.054762 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:44.821958 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:44.821787 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:45.510194 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:45.510154 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:46.251072 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:46.251038 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:46.990966 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:46.990928 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:47.795451 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:47.795378 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:48.631293 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:48.631247 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:48.678257 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:48.678229 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-26a128-846fd4d8c6-z4mcb" Oct 30 22:10:49.575244 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:49.575192 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:50.326049 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:50.325357 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:50.480713 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:50.480677 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:50.628366 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:50.628323 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:51.132261 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:51.132218 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:51.786479 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:51.784599 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:52.433995 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:52.433939 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:53.192061 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:53.192029 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:53.815751 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:53.815717 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:53.962013 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:53.961978 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:54.109505 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:54.109474 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:54.258102 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:54.258066 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd"] Oct 30 22:10:54.258634 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:54.258299 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd" podUID="1817b106-b8fd-4989-977b-115eca7245ec" containerName="istio-proxy" containerID="cri-o://7e47218a45cb68904c8dfe0dfa9e739d9e105bfc58f1b8af7248e2ad6717aab3" gracePeriod=30 Oct 30 22:10:54.261169 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:54.261144 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr"] Oct 30 22:10:54.261431 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:54.261402 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr" podUID="35de2b6b-4247-488f-baa1-182d56e82253" containerName="istio-proxy" containerID="cri-o://7b8aa45e3b0fe1435e9a7258dac91ffef05874b6c03fce67a56ea0869d3cd7fa" gracePeriod=30 Oct 30 22:10:54.628445 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:54.627578 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4fhws"] Oct 30 22:10:54.628445 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:54.627811 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4fhws" podUID="3e578cfa-7359-4c08-a8d0-56989d62227d" containerName="istio-proxy" containerID="cri-o://3fd6d374f6e0e9ab049c49320a0c11e1f964feb2dad0c1c081b6e4c3c69208f2" gracePeriod=30 Oct 30 22:10:54.685846 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:54.685815 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:54.835928 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:54.835891 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:55.357416 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:55.350305 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:56.089433 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:56.083363 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:56.231656 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:56.231622 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:56.393801 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:56.393768 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:56.724050 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:56.723953 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:56.893086 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:56.893005 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:57.481328 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:57.481297 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:58.402424 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:58.402094 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:58.779002 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:58.778790 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:59.231827 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:59.231789 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:59.429701 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:59.429666 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:59.604297 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:59.590667 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr" Oct 30 22:10:59.664358 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:59.664328 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd" Oct 30 22:10:59.730076 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:59.730040 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35de2b6b-4247-488f-baa1-182d56e82253-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35de2b6b-4247-488f-baa1-182d56e82253-workload-socket\") pod \"35de2b6b-4247-488f-baa1-182d56e82253\" (UID: \"35de2b6b-4247-488f-baa1-182d56e82253\") " Oct 30 22:10:59.730311 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:59.730122 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35de2b6b-4247-488f-baa1-182d56e82253-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35de2b6b-4247-488f-baa1-182d56e82253-credential-socket\") pod \"35de2b6b-4247-488f-baa1-182d56e82253\" (UID: \"35de2b6b-4247-488f-baa1-182d56e82253\") " Oct 30 22:10:59.730311 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:59.730166 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35de2b6b-4247-488f-baa1-182d56e82253-kube-api-access-fjt45\" (UniqueName: \"kubernetes.io/projected/35de2b6b-4247-488f-baa1-182d56e82253-kube-api-access-fjt45\") pod \"35de2b6b-4247-488f-baa1-182d56e82253\" (UID: \"35de2b6b-4247-488f-baa1-182d56e82253\") " Oct 30 22:10:59.730311 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:59.730189 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35de2b6b-4247-488f-baa1-182d56e82253-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35de2b6b-4247-488f-baa1-182d56e82253-istio-envoy\") pod \"35de2b6b-4247-488f-baa1-182d56e82253\" (UID: \"35de2b6b-4247-488f-baa1-182d56e82253\") " Oct 30 22:10:59.730311 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:59.730230 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35de2b6b-4247-488f-baa1-182d56e82253-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35de2b6b-4247-488f-baa1-182d56e82253-istiod-ca-cert\") pod \"35de2b6b-4247-488f-baa1-182d56e82253\" (UID: \"35de2b6b-4247-488f-baa1-182d56e82253\") " Oct 30 22:10:59.730311 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:59.730277 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/35de2b6b-4247-488f-baa1-182d56e82253-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35de2b6b-4247-488f-baa1-182d56e82253-istio-podinfo\") pod \"35de2b6b-4247-488f-baa1-182d56e82253\" (UID: \"35de2b6b-4247-488f-baa1-182d56e82253\") " Oct 30 22:10:59.732771 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:59.730378 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35de2b6b-4247-488f-baa1-182d56e82253-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35de2b6b-4247-488f-baa1-182d56e82253-workload-certs\") pod \"35de2b6b-4247-488f-baa1-182d56e82253\" (UID: \"35de2b6b-4247-488f-baa1-182d56e82253\") " Oct 30 22:10:59.732771 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:59.730567 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35de2b6b-4247-488f-baa1-182d56e82253-credential-socket" pod "35de2b6b-4247-488f-baa1-182d56e82253" (UID: "35de2b6b-4247-488f-baa1-182d56e82253"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:59.732771 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:59.730719 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35de2b6b-4247-488f-baa1-182d56e82253-workload-socket" pod "35de2b6b-4247-488f-baa1-182d56e82253" (UID: "35de2b6b-4247-488f-baa1-182d56e82253"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:59.735375 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:59.735348 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/35de2b6b-4247-488f-baa1-182d56e82253-istio-podinfo" pod "35de2b6b-4247-488f-baa1-182d56e82253" (UID: "35de2b6b-4247-488f-baa1-182d56e82253"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:10:59.735670 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:59.735648 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35de2b6b-4247-488f-baa1-182d56e82253-workload-certs" pod "35de2b6b-4247-488f-baa1-182d56e82253" (UID: "35de2b6b-4247-488f-baa1-182d56e82253"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:59.736626 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:59.736559 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35de2b6b-4247-488f-baa1-182d56e82253-istiod-ca-cert" pod "35de2b6b-4247-488f-baa1-182d56e82253" (UID: "35de2b6b-4247-488f-baa1-182d56e82253"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:10:59.736894 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:59.736781 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35de2b6b-4247-488f-baa1-182d56e82253-istio-token\" (UniqueName: \"kubernetes.io/projected/35de2b6b-4247-488f-baa1-182d56e82253-istio-token\") pod \"35de2b6b-4247-488f-baa1-182d56e82253\" (UID: \"35de2b6b-4247-488f-baa1-182d56e82253\") " Oct 30 22:10:59.736894 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:59.736823 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35de2b6b-4247-488f-baa1-182d56e82253-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35de2b6b-4247-488f-baa1-182d56e82253-istio-data\") pod \"35de2b6b-4247-488f-baa1-182d56e82253\" (UID: \"35de2b6b-4247-488f-baa1-182d56e82253\") " Oct 30 22:10:59.737724 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:59.737698 6720 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35de2b6b-4247-488f-baa1-182d56e82253-workload-certs\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:59.737901 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:59.737731 6720 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35de2b6b-4247-488f-baa1-182d56e82253-workload-socket\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:59.737901 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:59.737746 6720 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35de2b6b-4247-488f-baa1-182d56e82253-credential-socket\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:59.737901 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:59.737760 6720 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35de2b6b-4247-488f-baa1-182d56e82253-istiod-ca-cert\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:59.737901 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:59.737776 6720 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35de2b6b-4247-488f-baa1-182d56e82253-istio-podinfo\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:59.738135 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:59.738055 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35de2b6b-4247-488f-baa1-182d56e82253-istio-data" pod "35de2b6b-4247-488f-baa1-182d56e82253" (UID: "35de2b6b-4247-488f-baa1-182d56e82253"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:59.739328 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:59.739294 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35de2b6b-4247-488f-baa1-182d56e82253-istio-envoy" pod "35de2b6b-4247-488f-baa1-182d56e82253" (UID: "35de2b6b-4247-488f-baa1-182d56e82253"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:59.741527 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:59.741499 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35de2b6b-4247-488f-baa1-182d56e82253-istio-token" pod "35de2b6b-4247-488f-baa1-182d56e82253" (UID: "35de2b6b-4247-488f-baa1-182d56e82253"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:10:59.745136 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:59.744954 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35de2b6b-4247-488f-baa1-182d56e82253-kube-api-access-fjt45" pod "35de2b6b-4247-488f-baa1-182d56e82253" (UID: "35de2b6b-4247-488f-baa1-182d56e82253"). InnerVolumeSpecName "kube-api-access-fjt45". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:10:59.782514 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:59.782478 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:10:59.850411 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:59.845873 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1817b106-b8fd-4989-977b-115eca7245ec-kube-api-access-nv5pp\" (UniqueName: \"kubernetes.io/projected/1817b106-b8fd-4989-977b-115eca7245ec-kube-api-access-nv5pp\") pod \"1817b106-b8fd-4989-977b-115eca7245ec\" (UID: \"1817b106-b8fd-4989-977b-115eca7245ec\") " Oct 30 22:10:59.850411 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:59.846114 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1817b106-b8fd-4989-977b-115eca7245ec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1817b106-b8fd-4989-977b-115eca7245ec-istiod-ca-cert\") pod \"1817b106-b8fd-4989-977b-115eca7245ec\" (UID: \"1817b106-b8fd-4989-977b-115eca7245ec\") " Oct 30 22:10:59.850411 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:59.846221 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1817b106-b8fd-4989-977b-115eca7245ec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1817b106-b8fd-4989-977b-115eca7245ec-istio-data\") pod \"1817b106-b8fd-4989-977b-115eca7245ec\" (UID: \"1817b106-b8fd-4989-977b-115eca7245ec\") " Oct 30 22:10:59.850411 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:59.846263 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1817b106-b8fd-4989-977b-115eca7245ec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1817b106-b8fd-4989-977b-115eca7245ec-workload-socket\") pod \"1817b106-b8fd-4989-977b-115eca7245ec\" (UID: \"1817b106-b8fd-4989-977b-115eca7245ec\") " Oct 30 22:10:59.850411 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:59.846309 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1817b106-b8fd-4989-977b-115eca7245ec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1817b106-b8fd-4989-977b-115eca7245ec-credential-socket\") pod \"1817b106-b8fd-4989-977b-115eca7245ec\" (UID: \"1817b106-b8fd-4989-977b-115eca7245ec\") " Oct 30 22:10:59.850411 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:59.846344 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1817b106-b8fd-4989-977b-115eca7245ec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1817b106-b8fd-4989-977b-115eca7245ec-istio-envoy\") pod \"1817b106-b8fd-4989-977b-115eca7245ec\" (UID: \"1817b106-b8fd-4989-977b-115eca7245ec\") " Oct 30 22:10:59.850411 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:59.846374 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1817b106-b8fd-4989-977b-115eca7245ec-istio-token\" (UniqueName: \"kubernetes.io/projected/1817b106-b8fd-4989-977b-115eca7245ec-istio-token\") pod \"1817b106-b8fd-4989-977b-115eca7245ec\" (UID: \"1817b106-b8fd-4989-977b-115eca7245ec\") " Oct 30 22:10:59.850411 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:59.846427 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1817b106-b8fd-4989-977b-115eca7245ec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1817b106-b8fd-4989-977b-115eca7245ec-istio-podinfo\") pod \"1817b106-b8fd-4989-977b-115eca7245ec\" (UID: \"1817b106-b8fd-4989-977b-115eca7245ec\") " Oct 30 22:10:59.850411 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:59.846533 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1817b106-b8fd-4989-977b-115eca7245ec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1817b106-b8fd-4989-977b-115eca7245ec-workload-certs\") pod \"1817b106-b8fd-4989-977b-115eca7245ec\" (UID: \"1817b106-b8fd-4989-977b-115eca7245ec\") " Oct 30 22:10:59.850411 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:59.849526 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1817b106-b8fd-4989-977b-115eca7245ec-workload-certs" pod "1817b106-b8fd-4989-977b-115eca7245ec" (UID: "1817b106-b8fd-4989-977b-115eca7245ec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:59.866693 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:59.866406 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1817b106-b8fd-4989-977b-115eca7245ec-kube-api-access-nv5pp" pod "1817b106-b8fd-4989-977b-115eca7245ec" (UID: "1817b106-b8fd-4989-977b-115eca7245ec"). InnerVolumeSpecName "kube-api-access-nv5pp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:10:59.867583 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:59.867559 6720 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35de2b6b-4247-488f-baa1-182d56e82253-istio-token\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:59.867735 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:59.867721 6720 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1817b106-b8fd-4989-977b-115eca7245ec-workload-certs\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:59.867820 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:59.867810 6720 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35de2b6b-4247-488f-baa1-182d56e82253-istio-data\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:59.867895 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:59.867883 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nv5pp\" (UniqueName: \"kubernetes.io/projected/1817b106-b8fd-4989-977b-115eca7245ec-kube-api-access-nv5pp\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:59.867966 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:59.867956 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjt45\" (UniqueName: \"kubernetes.io/projected/35de2b6b-4247-488f-baa1-182d56e82253-kube-api-access-fjt45\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:59.868040 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:59.868031 6720 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35de2b6b-4247-488f-baa1-182d56e82253-istio-envoy\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:59.868373 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:59.868354 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1817b106-b8fd-4989-977b-115eca7245ec-istio-data" pod "1817b106-b8fd-4989-977b-115eca7245ec" (UID: "1817b106-b8fd-4989-977b-115eca7245ec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:59.868701 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:59.868665 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1817b106-b8fd-4989-977b-115eca7245ec-workload-socket" pod "1817b106-b8fd-4989-977b-115eca7245ec" (UID: "1817b106-b8fd-4989-977b-115eca7245ec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:59.868790 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:59.868747 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1817b106-b8fd-4989-977b-115eca7245ec-credential-socket" pod "1817b106-b8fd-4989-977b-115eca7245ec" (UID: "1817b106-b8fd-4989-977b-115eca7245ec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:59.868928 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:59.868903 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1817b106-b8fd-4989-977b-115eca7245ec-istiod-ca-cert" pod "1817b106-b8fd-4989-977b-115eca7245ec" (UID: "1817b106-b8fd-4989-977b-115eca7245ec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:10:59.875217 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:59.875047 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1817b106-b8fd-4989-977b-115eca7245ec-istio-token" pod "1817b106-b8fd-4989-977b-115eca7245ec" (UID: "1817b106-b8fd-4989-977b-115eca7245ec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:10:59.896412 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:59.891339 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1817b106-b8fd-4989-977b-115eca7245ec-istio-envoy" pod "1817b106-b8fd-4989-977b-115eca7245ec" (UID: "1817b106-b8fd-4989-977b-115eca7245ec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:59.896412 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:59.891547 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1817b106-b8fd-4989-977b-115eca7245ec-istio-podinfo" pod "1817b106-b8fd-4989-977b-115eca7245ec" (UID: "1817b106-b8fd-4989-977b-115eca7245ec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:10:59.928411 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:59.923977 6720 generic.go:356] "Generic (PLEG): container finished" podID="1817b106-b8fd-4989-977b-115eca7245ec" containerID="7e47218a45cb68904c8dfe0dfa9e739d9e105bfc58f1b8af7248e2ad6717aab3" exitCode=0 Oct 30 22:10:59.928411 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:59.924089 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd" event={"ID":"1817b106-b8fd-4989-977b-115eca7245ec","Type":"ContainerDied","Data":"7e47218a45cb68904c8dfe0dfa9e739d9e105bfc58f1b8af7248e2ad6717aab3"} Oct 30 22:10:59.928411 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:59.924144 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd" event={"ID":"1817b106-b8fd-4989-977b-115eca7245ec","Type":"ContainerDied","Data":"448525bab286267e657f7b9923b4d716a8e09d1d342b94ccd55571e5a44cc6fd"} Oct 30 22:10:59.928411 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:59.924173 6720 scope.go:122] "RemoveContainer" containerID="7e47218a45cb68904c8dfe0dfa9e739d9e105bfc58f1b8af7248e2ad6717aab3" Oct 30 22:10:59.928411 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:59.924336 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd" Oct 30 22:10:59.975606 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:59.974449 6720 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1817b106-b8fd-4989-977b-115eca7245ec-istiod-ca-cert\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:59.975606 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:59.974513 6720 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1817b106-b8fd-4989-977b-115eca7245ec-istio-data\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:59.975606 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:59.974532 6720 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1817b106-b8fd-4989-977b-115eca7245ec-workload-socket\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:59.975606 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:59.974546 6720 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1817b106-b8fd-4989-977b-115eca7245ec-credential-socket\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:59.975606 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:59.974561 6720 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1817b106-b8fd-4989-977b-115eca7245ec-istio-envoy\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:59.975606 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:59.974574 6720 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1817b106-b8fd-4989-977b-115eca7245ec-istio-token\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:59.975606 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:59.974589 6720 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1817b106-b8fd-4989-977b-115eca7245ec-istio-podinfo\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:59.986405 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:59.986347 6720 generic.go:356] "Generic (PLEG): container finished" podID="35de2b6b-4247-488f-baa1-182d56e82253" containerID="7b8aa45e3b0fe1435e9a7258dac91ffef05874b6c03fce67a56ea0869d3cd7fa" exitCode=0 Oct 30 22:10:59.986594 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:10:59.986490 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr" Oct 30 22:11:00.020927 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:00.015069 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr" event={"ID":"35de2b6b-4247-488f-baa1-182d56e82253","Type":"ContainerDied","Data":"7b8aa45e3b0fe1435e9a7258dac91ffef05874b6c03fce67a56ea0869d3cd7fa"} Oct 30 22:11:00.020927 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:00.015139 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr" event={"ID":"35de2b6b-4247-488f-baa1-182d56e82253","Type":"ContainerDied","Data":"dccbccc892c1ed30e8b7af8c7f2bd29bc99514cbab50ebbd6ec7002b17b45eda"} Oct 30 22:11:00.037139 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:00.037018 6720 scope.go:122] "RemoveContainer" containerID="7e47218a45cb68904c8dfe0dfa9e739d9e105bfc58f1b8af7248e2ad6717aab3" Oct 30 22:11:00.037357 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:00.037330 6720 generic.go:356] "Generic (PLEG): container finished" podID="3e578cfa-7359-4c08-a8d0-56989d62227d" containerID="3fd6d374f6e0e9ab049c49320a0c11e1f964feb2dad0c1c081b6e4c3c69208f2" exitCode=0 Oct 30 22:11:00.037475 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:00.037432 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4fhws" event={"ID":"3e578cfa-7359-4c08-a8d0-56989d62227d","Type":"ContainerDied","Data":"3fd6d374f6e0e9ab049c49320a0c11e1f964feb2dad0c1c081b6e4c3c69208f2"} Oct 30 22:11:00.037793 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:11:00.037766 6720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e47218a45cb68904c8dfe0dfa9e739d9e105bfc58f1b8af7248e2ad6717aab3\": container with ID starting with 7e47218a45cb68904c8dfe0dfa9e739d9e105bfc58f1b8af7248e2ad6717aab3 not found: ID does not exist" containerID="7e47218a45cb68904c8dfe0dfa9e739d9e105bfc58f1b8af7248e2ad6717aab3" Oct 30 22:11:00.037869 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:00.037806 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e47218a45cb68904c8dfe0dfa9e739d9e105bfc58f1b8af7248e2ad6717aab3"} err="failed to get container status \"7e47218a45cb68904c8dfe0dfa9e739d9e105bfc58f1b8af7248e2ad6717aab3\": rpc error: code = NotFound desc = could not find container \"7e47218a45cb68904c8dfe0dfa9e739d9e105bfc58f1b8af7248e2ad6717aab3\": container with ID starting with 7e47218a45cb68904c8dfe0dfa9e739d9e105bfc58f1b8af7248e2ad6717aab3 not found: ID does not exist" Oct 30 22:11:00.037869 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:00.037828 6720 scope.go:122] "RemoveContainer" containerID="7b8aa45e3b0fe1435e9a7258dac91ffef05874b6c03fce67a56ea0869d3cd7fa" Oct 30 22:11:00.068123 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:00.068098 6720 scope.go:122] "RemoveContainer" containerID="7b8aa45e3b0fe1435e9a7258dac91ffef05874b6c03fce67a56ea0869d3cd7fa" Oct 30 22:11:00.069050 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:11:00.068825 6720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b8aa45e3b0fe1435e9a7258dac91ffef05874b6c03fce67a56ea0869d3cd7fa\": container with ID starting with 7b8aa45e3b0fe1435e9a7258dac91ffef05874b6c03fce67a56ea0869d3cd7fa not found: ID does not exist" containerID="7b8aa45e3b0fe1435e9a7258dac91ffef05874b6c03fce67a56ea0869d3cd7fa" Oct 30 22:11:00.069050 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:00.068863 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b8aa45e3b0fe1435e9a7258dac91ffef05874b6c03fce67a56ea0869d3cd7fa"} err="failed to get container status \"7b8aa45e3b0fe1435e9a7258dac91ffef05874b6c03fce67a56ea0869d3cd7fa\": rpc error: code = NotFound desc = could not find container \"7b8aa45e3b0fe1435e9a7258dac91ffef05874b6c03fce67a56ea0869d3cd7fa\": container with ID starting with 7b8aa45e3b0fe1435e9a7258dac91ffef05874b6c03fce67a56ea0869d3cd7fa not found: ID does not exist" Oct 30 22:11:00.070144 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:00.069733 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd"] Oct 30 22:11:00.074452 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:00.074352 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-r4rcd"] Oct 30 22:11:00.080284 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:11:00.080243 6720 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1817b106_b8fd_4989_977b_115eca7245ec.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35de2b6b_4247_488f_baa1_182d56e82253.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1817b106_b8fd_4989_977b_115eca7245ec.slice/crio-448525bab286267e657f7b9923b4d716a8e09d1d342b94ccd55571e5a44cc6fd\": RecentStats: unable to find data in memory cache]" Oct 30 22:11:00.080446 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:11:00.080309 6720 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1817b106_b8fd_4989_977b_115eca7245ec.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1817b106_b8fd_4989_977b_115eca7245ec.slice/crio-448525bab286267e657f7b9923b4d716a8e09d1d342b94ccd55571e5a44cc6fd\": RecentStats: unable to find data in memory cache]" Oct 30 22:11:00.133683 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:00.133648 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr"] Oct 30 22:11:00.149624 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:00.148914 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-gh7dr"] Oct 30 22:11:00.355192 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:00.330338 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:00.383830 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:00.383789 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4fhws" Oct 30 22:11:00.520231 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:00.520193 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:00.586654 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:00.586625 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e578cfa-7359-4c08-a8d0-56989d62227d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e578cfa-7359-4c08-a8d0-56989d62227d-istio-envoy\") pod \"3e578cfa-7359-4c08-a8d0-56989d62227d\" (UID: \"3e578cfa-7359-4c08-a8d0-56989d62227d\") " Oct 30 22:11:00.586872 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:00.586695 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e578cfa-7359-4c08-a8d0-56989d62227d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e578cfa-7359-4c08-a8d0-56989d62227d-workload-certs\") pod \"3e578cfa-7359-4c08-a8d0-56989d62227d\" (UID: \"3e578cfa-7359-4c08-a8d0-56989d62227d\") " Oct 30 22:11:00.586872 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:00.586769 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e578cfa-7359-4c08-a8d0-56989d62227d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e578cfa-7359-4c08-a8d0-56989d62227d-credential-socket\") pod \"3e578cfa-7359-4c08-a8d0-56989d62227d\" (UID: \"3e578cfa-7359-4c08-a8d0-56989d62227d\") " Oct 30 22:11:00.586872 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:00.586833 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3e578cfa-7359-4c08-a8d0-56989d62227d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e578cfa-7359-4c08-a8d0-56989d62227d-istio-podinfo\") pod \"3e578cfa-7359-4c08-a8d0-56989d62227d\" (UID: \"3e578cfa-7359-4c08-a8d0-56989d62227d\") " Oct 30 22:11:00.587051 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:00.586895 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e578cfa-7359-4c08-a8d0-56989d62227d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e578cfa-7359-4c08-a8d0-56989d62227d-workload-socket\") pod \"3e578cfa-7359-4c08-a8d0-56989d62227d\" (UID: \"3e578cfa-7359-4c08-a8d0-56989d62227d\") " Oct 30 22:11:00.587051 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:00.586933 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e578cfa-7359-4c08-a8d0-56989d62227d-istio-token\" (UniqueName: \"kubernetes.io/projected/3e578cfa-7359-4c08-a8d0-56989d62227d-istio-token\") pod \"3e578cfa-7359-4c08-a8d0-56989d62227d\" (UID: \"3e578cfa-7359-4c08-a8d0-56989d62227d\") " Oct 30 22:11:00.587051 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:00.586952 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e578cfa-7359-4c08-a8d0-56989d62227d-workload-certs" pod "3e578cfa-7359-4c08-a8d0-56989d62227d" (UID: "3e578cfa-7359-4c08-a8d0-56989d62227d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:00.587051 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:00.587027 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e578cfa-7359-4c08-a8d0-56989d62227d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e578cfa-7359-4c08-a8d0-56989d62227d-istiod-ca-cert\") pod \"3e578cfa-7359-4c08-a8d0-56989d62227d\" (UID: \"3e578cfa-7359-4c08-a8d0-56989d62227d\") " Oct 30 22:11:00.587253 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:00.587082 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e578cfa-7359-4c08-a8d0-56989d62227d-credential-socket" pod "3e578cfa-7359-4c08-a8d0-56989d62227d" (UID: "3e578cfa-7359-4c08-a8d0-56989d62227d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:00.587253 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:00.587113 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e578cfa-7359-4c08-a8d0-56989d62227d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e578cfa-7359-4c08-a8d0-56989d62227d-istio-data\") pod \"3e578cfa-7359-4c08-a8d0-56989d62227d\" (UID: \"3e578cfa-7359-4c08-a8d0-56989d62227d\") " Oct 30 22:11:00.587253 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:00.587153 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e578cfa-7359-4c08-a8d0-56989d62227d-kube-api-access-hfkn2\" (UniqueName: \"kubernetes.io/projected/3e578cfa-7359-4c08-a8d0-56989d62227d-kube-api-access-hfkn2\") pod \"3e578cfa-7359-4c08-a8d0-56989d62227d\" (UID: \"3e578cfa-7359-4c08-a8d0-56989d62227d\") " Oct 30 22:11:00.587253 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:00.587153 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e578cfa-7359-4c08-a8d0-56989d62227d-workload-socket" pod "3e578cfa-7359-4c08-a8d0-56989d62227d" (UID: "3e578cfa-7359-4c08-a8d0-56989d62227d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:00.587532 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:00.587403 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e578cfa-7359-4c08-a8d0-56989d62227d-istio-data" pod "3e578cfa-7359-4c08-a8d0-56989d62227d" (UID: "3e578cfa-7359-4c08-a8d0-56989d62227d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:00.587532 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:00.587480 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e578cfa-7359-4c08-a8d0-56989d62227d-istiod-ca-cert" pod "3e578cfa-7359-4c08-a8d0-56989d62227d" (UID: "3e578cfa-7359-4c08-a8d0-56989d62227d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:11:00.587942 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:00.587751 6720 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e578cfa-7359-4c08-a8d0-56989d62227d-workload-certs\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:11:00.587942 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:00.587779 6720 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e578cfa-7359-4c08-a8d0-56989d62227d-credential-socket\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:11:00.587942 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:00.587794 6720 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e578cfa-7359-4c08-a8d0-56989d62227d-workload-socket\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:11:00.587942 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:00.587808 6720 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e578cfa-7359-4c08-a8d0-56989d62227d-istiod-ca-cert\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:11:00.587942 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:00.587822 6720 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e578cfa-7359-4c08-a8d0-56989d62227d-istio-data\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:11:00.590049 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:00.590022 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e578cfa-7359-4c08-a8d0-56989d62227d-istio-envoy" pod "3e578cfa-7359-4c08-a8d0-56989d62227d" (UID: "3e578cfa-7359-4c08-a8d0-56989d62227d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:11:00.590177 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:00.590125 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e578cfa-7359-4c08-a8d0-56989d62227d-kube-api-access-hfkn2" pod "3e578cfa-7359-4c08-a8d0-56989d62227d" (UID: "3e578cfa-7359-4c08-a8d0-56989d62227d"). InnerVolumeSpecName "kube-api-access-hfkn2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:11:00.590177 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:00.590157 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3e578cfa-7359-4c08-a8d0-56989d62227d-istio-podinfo" pod "3e578cfa-7359-4c08-a8d0-56989d62227d" (UID: "3e578cfa-7359-4c08-a8d0-56989d62227d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:11:00.592510 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:00.592487 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e578cfa-7359-4c08-a8d0-56989d62227d-istio-token" pod "3e578cfa-7359-4c08-a8d0-56989d62227d" (UID: "3e578cfa-7359-4c08-a8d0-56989d62227d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:11:00.704852 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:00.704683 6720 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e578cfa-7359-4c08-a8d0-56989d62227d-istio-token\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:11:00.704852 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:00.704725 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfkn2\" (UniqueName: \"kubernetes.io/projected/3e578cfa-7359-4c08-a8d0-56989d62227d-kube-api-access-hfkn2\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:11:00.704852 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:00.704741 6720 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e578cfa-7359-4c08-a8d0-56989d62227d-istio-envoy\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:11:00.704852 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:00.704755 6720 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e578cfa-7359-4c08-a8d0-56989d62227d-istio-podinfo\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:11:00.719869 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:00.718947 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:00.917433 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:00.917376 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:01.045920 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:01.045830 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4fhws" Oct 30 22:11:01.045920 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:01.045854 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4fhws" event={"ID":"3e578cfa-7359-4c08-a8d0-56989d62227d","Type":"ContainerDied","Data":"0e86e59d82f19205fb9d83a1588a39aa78a7f1ba966b286e89c99b3d04d84ee2"} Oct 30 22:11:01.045920 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:01.045913 6720 scope.go:122] "RemoveContainer" containerID="3fd6d374f6e0e9ab049c49320a0c11e1f964feb2dad0c1c081b6e4c3c69208f2" Oct 30 22:11:01.102090 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:01.102058 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4fhws"] Oct 30 22:11:01.107611 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:01.107586 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4fhws"] Oct 30 22:11:01.871964 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:01.871932 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:01.961059 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:01.960936 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1817b106-b8fd-4989-977b-115eca7245ec" path="/var/lib/kubelet/pods/1817b106-b8fd-4989-977b-115eca7245ec/volumes" Oct 30 22:11:01.961732 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:01.961704 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35de2b6b-4247-488f-baa1-182d56e82253" path="/var/lib/kubelet/pods/35de2b6b-4247-488f-baa1-182d56e82253/volumes" Oct 30 22:11:01.962324 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:01.962304 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e578cfa-7359-4c08-a8d0-56989d62227d" path="/var/lib/kubelet/pods/3e578cfa-7359-4c08-a8d0-56989d62227d/volumes" Oct 30 22:11:02.051079 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:02.051046 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:02.217464 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:02.217095 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:02.381331 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:02.381296 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:02.624896 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:02.624862 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:02.800782 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:02.800743 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:03.146445 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:03.143191 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:03.371527 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:03.371489 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:03.524464 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:03.522575 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:03.686408 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:03.682849 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:03.848465 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:03.847458 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:04.469093 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:04.469057 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:04.616176 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:04.616136 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:05.390441 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:05.371002 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:05.524651 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:05.524601 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:05.677176 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:05.676552 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:06.711966 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:06.711928 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:07.477348 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:07.477314 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:08.349351 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:08.349279 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:09.033316 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:09.033279 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:09.804820 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:09.804776 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:10.404045 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:11:10.404006 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 22:11:10.648636 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:10.648517 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:10.845128 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:10.845038 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:11.419808 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:11.419770 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:12.109787 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:12.109752 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:12.258934 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:12.258901 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:12.551511 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:12.551362 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:12.940431 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:12.940353 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:13.146626 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:13.146590 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:13.833495 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:13.833458 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:14.698239 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:14.698196 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:15.326469 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:15.326431 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:15.502188 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:15.502157 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:16.223807 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:16.223774 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:16.540322 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:16.540227 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:16.695562 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:16.695472 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:16.849346 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:16.849252 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:17.167297 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:17.167267 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:17.425955 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:17.425877 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:17.839735 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:17.839654 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:18.000410 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:18.000248 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:18.535859 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:18.535830 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:18.763847 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:18.763818 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:19.440973 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:19.440942 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-a1ea1e-5b7bb88c4b-f6w5x"] Oct 30 22:11:19.441627 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:19.441612 6720 cpu_manager.go:414] "containerMap: removing container" podUID="1817b106-b8fd-4989-977b-115eca7245ec" containerName="istio-proxy" Oct 30 22:11:19.441683 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:19.441628 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1817b106-b8fd-4989-977b-115eca7245ec" containerName="istio-proxy" Oct 30 22:11:19.441683 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:19.441647 6720 cpu_manager.go:414] "containerMap: removing container" podUID="35de2b6b-4247-488f-baa1-182d56e82253" containerName="istio-proxy" Oct 30 22:11:19.441683 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:19.441652 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35de2b6b-4247-488f-baa1-182d56e82253" containerName="istio-proxy" Oct 30 22:11:19.441683 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:19.441672 6720 cpu_manager.go:414] "containerMap: removing container" podUID="3e578cfa-7359-4c08-a8d0-56989d62227d" containerName="istio-proxy" Oct 30 22:11:19.441683 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:19.441678 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e578cfa-7359-4c08-a8d0-56989d62227d" containerName="istio-proxy" Oct 30 22:11:19.441831 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:19.441781 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e578cfa-7359-4c08-a8d0-56989d62227d" containerName="istio-proxy" Oct 30 22:11:19.441831 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:19.441795 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="1817b106-b8fd-4989-977b-115eca7245ec" containerName="istio-proxy" Oct 30 22:11:19.441831 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:19.441802 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="35de2b6b-4247-488f-baa1-182d56e82253" containerName="istio-proxy" Oct 30 22:11:19.445264 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:19.445246 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a1ea1e-5b7bb88c4b-f6w5x" Oct 30 22:11:19.451692 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:19.451665 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-a1ea1e-serving-cert\"" Oct 30 22:11:19.461630 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:19.461588 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-a1ea1e-5b7bb88c4b-f6w5x"] Oct 30 22:11:19.518529 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:19.518492 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/083d25b4-5991-4298-b2f3-392d28c4e620-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-a1ea1e-5b7bb88c4b-f6w5x\" (UID: \"083d25b4-5991-4298-b2f3-392d28c4e620\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a1ea1e-5b7bb88c4b-f6w5x" Oct 30 22:11:19.518715 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:19.518651 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/083d25b4-5991-4298-b2f3-392d28c4e620-maas-api-tls\") pod \"maas-api-e2e-extmod-b-a1ea1e-5b7bb88c4b-f6w5x\" (UID: \"083d25b4-5991-4298-b2f3-392d28c4e620\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a1ea1e-5b7bb88c4b-f6w5x" Oct 30 22:11:19.518905 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:19.518885 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/083d25b4-5991-4298-b2f3-392d28c4e620-metrics-certs\") pod \"maas-api-e2e-extmod-b-a1ea1e-5b7bb88c4b-f6w5x\" (UID: \"083d25b4-5991-4298-b2f3-392d28c4e620\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a1ea1e-5b7bb88c4b-f6w5x" Oct 30 22:11:19.519025 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:19.519000 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svmmm\" (UniqueName: \"kubernetes.io/projected/083d25b4-5991-4298-b2f3-392d28c4e620-kube-api-access-svmmm\") pod \"maas-api-e2e-extmod-b-a1ea1e-5b7bb88c4b-f6w5x\" (UID: \"083d25b4-5991-4298-b2f3-392d28c4e620\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a1ea1e-5b7bb88c4b-f6w5x" Oct 30 22:11:19.523483 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:19.523437 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:19.628751 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:19.628715 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/083d25b4-5991-4298-b2f3-392d28c4e620-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-a1ea1e-5b7bb88c4b-f6w5x\" (UID: \"083d25b4-5991-4298-b2f3-392d28c4e620\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a1ea1e-5b7bb88c4b-f6w5x" Oct 30 22:11:19.629224 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:19.628876 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/083d25b4-5991-4298-b2f3-392d28c4e620-maas-api-tls\") pod \"maas-api-e2e-extmod-b-a1ea1e-5b7bb88c4b-f6w5x\" (UID: \"083d25b4-5991-4298-b2f3-392d28c4e620\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a1ea1e-5b7bb88c4b-f6w5x" Oct 30 22:11:19.629224 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:19.629064 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/083d25b4-5991-4298-b2f3-392d28c4e620-metrics-certs\") pod \"maas-api-e2e-extmod-b-a1ea1e-5b7bb88c4b-f6w5x\" (UID: \"083d25b4-5991-4298-b2f3-392d28c4e620\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a1ea1e-5b7bb88c4b-f6w5x" Oct 30 22:11:19.629224 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:19.629126 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svmmm\" (UniqueName: \"kubernetes.io/projected/083d25b4-5991-4298-b2f3-392d28c4e620-kube-api-access-svmmm\") pod \"maas-api-e2e-extmod-b-a1ea1e-5b7bb88c4b-f6w5x\" (UID: \"083d25b4-5991-4298-b2f3-392d28c4e620\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a1ea1e-5b7bb88c4b-f6w5x" Oct 30 22:11:19.630060 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:19.630031 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/083d25b4-5991-4298-b2f3-392d28c4e620-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-a1ea1e-5b7bb88c4b-f6w5x\" (UID: \"083d25b4-5991-4298-b2f3-392d28c4e620\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a1ea1e-5b7bb88c4b-f6w5x" Oct 30 22:11:19.632762 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:19.632734 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/083d25b4-5991-4298-b2f3-392d28c4e620-maas-api-tls\") pod \"maas-api-e2e-extmod-b-a1ea1e-5b7bb88c4b-f6w5x\" (UID: \"083d25b4-5991-4298-b2f3-392d28c4e620\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a1ea1e-5b7bb88c4b-f6w5x" Oct 30 22:11:19.635329 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:19.635301 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/083d25b4-5991-4298-b2f3-392d28c4e620-metrics-certs\") pod \"maas-api-e2e-extmod-b-a1ea1e-5b7bb88c4b-f6w5x\" (UID: \"083d25b4-5991-4298-b2f3-392d28c4e620\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a1ea1e-5b7bb88c4b-f6w5x" Oct 30 22:11:19.642785 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:19.642751 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svmmm\" (UniqueName: \"kubernetes.io/projected/083d25b4-5991-4298-b2f3-392d28c4e620-kube-api-access-svmmm\") pod \"maas-api-e2e-extmod-b-a1ea1e-5b7bb88c4b-f6w5x\" (UID: \"083d25b4-5991-4298-b2f3-392d28c4e620\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a1ea1e-5b7bb88c4b-f6w5x" Oct 30 22:11:19.672063 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:19.672005 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:19.759742 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:19.759654 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a1ea1e-5b7bb88c4b-f6w5x" Oct 30 22:11:20.058526 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:20.058400 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:20.112839 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:20.112812 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-a1ea1e-5b7bb88c4b-f6w5x"] Oct 30 22:11:20.115874 ip-10-0-137-181 kubenswrapper[6720]: W1030 22:11:20.115845 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod083d25b4_5991_4298_b2f3_392d28c4e620.slice/crio-d5a81d720abb8d2ee9dcd87cb196612cd2a11e5ff07de0cfa107dd7dd605c398 WatchSource:0}: Error finding container d5a81d720abb8d2ee9dcd87cb196612cd2a11e5ff07de0cfa107dd7dd605c398: Status 404 returned error can't find the container with id d5a81d720abb8d2ee9dcd87cb196612cd2a11e5ff07de0cfa107dd7dd605c398 Oct 30 22:11:20.203556 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:20.203522 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a1ea1e-5b7bb88c4b-f6w5x" event={"ID":"083d25b4-5991-4298-b2f3-392d28c4e620","Type":"ContainerStarted","Data":"d5a81d720abb8d2ee9dcd87cb196612cd2a11e5ff07de0cfa107dd7dd605c398"} Oct 30 22:11:20.379409 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:20.377272 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:20.642209 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:20.642175 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:21.211461 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:21.211420 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a1ea1e-5b7bb88c4b-f6w5x" event={"ID":"083d25b4-5991-4298-b2f3-392d28c4e620","Type":"ContainerStarted","Data":"b4e09fe63d07362925fd068e98d052f6e5bd5fdd45aa84498da411a31fde3ad4"} Oct 30 22:11:21.211683 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:21.211537 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a1ea1e-5b7bb88c4b-f6w5x" Oct 30 22:11:21.238105 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:21.238035 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a1ea1e-5b7bb88c4b-f6w5x" podStartSLOduration=1.7263842029999998 podStartE2EDuration="2.237996535s" totalImagesPullingTime="511.612332ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:21.234366139 +0000 UTC m=+1847.405331315" watchObservedRunningTime="2026-10-30 22:11:21.237996535 +0000 UTC m=+1847.408961752" Oct 30 22:11:21.589968 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:21.589850 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:22.206378 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:22.206338 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-f0f5e2-664fb64799-h7rnw"] Oct 30 22:11:22.211514 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:22.211489 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f0f5e2-664fb64799-h7rnw" Oct 30 22:11:22.221269 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:22.221239 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-f0f5e2-serving-cert\"" Oct 30 22:11:22.230167 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:22.230143 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-f0f5e2-664fb64799-h7rnw"] Oct 30 22:11:22.286929 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:22.286886 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9pxt\" (UniqueName: \"kubernetes.io/projected/2aa72ad3-ab65-436a-abd6-8bf4bda11bf3-kube-api-access-p9pxt\") pod \"maas-api-e2e-autores-f0f5e2-664fb64799-h7rnw\" (UID: \"2aa72ad3-ab65-436a-abd6-8bf4bda11bf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f0f5e2-664fb64799-h7rnw" Oct 30 22:11:22.287096 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:22.287049 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2aa72ad3-ab65-436a-abd6-8bf4bda11bf3-metrics-certs\") pod \"maas-api-e2e-autores-f0f5e2-664fb64799-h7rnw\" (UID: \"2aa72ad3-ab65-436a-abd6-8bf4bda11bf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f0f5e2-664fb64799-h7rnw" Oct 30 22:11:22.287161 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:22.287115 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2aa72ad3-ab65-436a-abd6-8bf4bda11bf3-openshift-service-ca\") pod \"maas-api-e2e-autores-f0f5e2-664fb64799-h7rnw\" (UID: \"2aa72ad3-ab65-436a-abd6-8bf4bda11bf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f0f5e2-664fb64799-h7rnw" Oct 30 22:11:22.287396 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:22.287357 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2aa72ad3-ab65-436a-abd6-8bf4bda11bf3-maas-api-tls\") pod \"maas-api-e2e-autores-f0f5e2-664fb64799-h7rnw\" (UID: \"2aa72ad3-ab65-436a-abd6-8bf4bda11bf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f0f5e2-664fb64799-h7rnw" Oct 30 22:11:22.316910 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:22.316870 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:22.388866 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:22.388834 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2aa72ad3-ab65-436a-abd6-8bf4bda11bf3-maas-api-tls\") pod \"maas-api-e2e-autores-f0f5e2-664fb64799-h7rnw\" (UID: \"2aa72ad3-ab65-436a-abd6-8bf4bda11bf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f0f5e2-664fb64799-h7rnw" Oct 30 22:11:22.389088 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:22.388928 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9pxt\" (UniqueName: \"kubernetes.io/projected/2aa72ad3-ab65-436a-abd6-8bf4bda11bf3-kube-api-access-p9pxt\") pod \"maas-api-e2e-autores-f0f5e2-664fb64799-h7rnw\" (UID: \"2aa72ad3-ab65-436a-abd6-8bf4bda11bf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f0f5e2-664fb64799-h7rnw" Oct 30 22:11:22.389088 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:22.389033 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2aa72ad3-ab65-436a-abd6-8bf4bda11bf3-metrics-certs\") pod \"maas-api-e2e-autores-f0f5e2-664fb64799-h7rnw\" (UID: \"2aa72ad3-ab65-436a-abd6-8bf4bda11bf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f0f5e2-664fb64799-h7rnw" Oct 30 22:11:22.389217 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:22.389086 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2aa72ad3-ab65-436a-abd6-8bf4bda11bf3-openshift-service-ca\") pod \"maas-api-e2e-autores-f0f5e2-664fb64799-h7rnw\" (UID: \"2aa72ad3-ab65-436a-abd6-8bf4bda11bf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f0f5e2-664fb64799-h7rnw" Oct 30 22:11:22.389792 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:22.389766 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2aa72ad3-ab65-436a-abd6-8bf4bda11bf3-openshift-service-ca\") pod \"maas-api-e2e-autores-f0f5e2-664fb64799-h7rnw\" (UID: \"2aa72ad3-ab65-436a-abd6-8bf4bda11bf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f0f5e2-664fb64799-h7rnw" Oct 30 22:11:22.392621 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:22.392598 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2aa72ad3-ab65-436a-abd6-8bf4bda11bf3-metrics-certs\") pod \"maas-api-e2e-autores-f0f5e2-664fb64799-h7rnw\" (UID: \"2aa72ad3-ab65-436a-abd6-8bf4bda11bf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f0f5e2-664fb64799-h7rnw" Oct 30 22:11:22.392727 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:22.392713 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2aa72ad3-ab65-436a-abd6-8bf4bda11bf3-maas-api-tls\") pod \"maas-api-e2e-autores-f0f5e2-664fb64799-h7rnw\" (UID: \"2aa72ad3-ab65-436a-abd6-8bf4bda11bf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f0f5e2-664fb64799-h7rnw" Oct 30 22:11:22.404323 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:22.404294 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9pxt\" (UniqueName: \"kubernetes.io/projected/2aa72ad3-ab65-436a-abd6-8bf4bda11bf3-kube-api-access-p9pxt\") pod \"maas-api-e2e-autores-f0f5e2-664fb64799-h7rnw\" (UID: \"2aa72ad3-ab65-436a-abd6-8bf4bda11bf3\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-f0f5e2-664fb64799-h7rnw" Oct 30 22:11:22.499881 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:22.499795 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:22.531232 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:22.531204 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f0f5e2-664fb64799-h7rnw" Oct 30 22:11:22.904767 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:22.896745 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-f0f5e2-664fb64799-h7rnw"] Oct 30 22:11:23.219197 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:23.219107 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:23.227456 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:23.227416 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f0f5e2-664fb64799-h7rnw" event={"ID":"2aa72ad3-ab65-436a-abd6-8bf4bda11bf3","Type":"ContainerStarted","Data":"27377e72dfc0eb17f64af71e41499c6802cdbb6ac42359d457e899c98353771a"} Oct 30 22:11:23.974053 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:23.974019 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:24.243670 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:24.243575 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f0f5e2-664fb64799-h7rnw" event={"ID":"2aa72ad3-ab65-436a-abd6-8bf4bda11bf3","Type":"ContainerStarted","Data":"d96a27a7e5b0aff876ebc829b91438ed6367ed10bf1c8c84e7a4c4a6da2431d1"} Oct 30 22:11:24.244119 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:24.243795 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f0f5e2-664fb64799-h7rnw" Oct 30 22:11:24.265522 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:24.265466 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f0f5e2-664fb64799-h7rnw" podStartSLOduration=1.81350382 podStartE2EDuration="2.265449227s" totalImagesPullingTime="451.945407ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:24.264987101 +0000 UTC m=+1850.435952283" watchObservedRunningTime="2026-10-30 22:11:24.265449227 +0000 UTC m=+1850.436414409" Oct 30 22:11:24.746109 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:24.746074 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:25.348571 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:25.347599 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:26.006744 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:26.006711 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:26.644923 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:26.644849 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:27.224799 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:27.224770 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a1ea1e-5b7bb88c4b-f6w5x" Oct 30 22:11:27.441084 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:27.441049 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:28.031939 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:28.031880 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:28.661057 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:28.661024 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:28.823690 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:28.823660 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:29.444960 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:29.444921 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:30.099807 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:30.099772 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:30.263515 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:30.263488 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f0f5e2-664fb64799-h7rnw" Oct 30 22:11:30.716251 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:30.716219 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:31.509354 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:31.509301 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:32.225713 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:32.225670 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:32.820731 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:32.820696 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:33.554590 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:33.554561 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:33.620755 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:33.620726 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-n4hgm"] Oct 30 22:11:33.624995 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:33.624974 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-n4hgm" Oct 30 22:11:33.635177 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:33.635142 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-5313f0-openshift-default-dockercfg-scpj8\"" Oct 30 22:11:33.651016 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:33.650983 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-n4hgm"] Oct 30 22:11:33.776291 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:33.776259 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fa901df-6086-43c3-8f7e-a7a633607486-workload-socket\") pod \"e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-n4hgm\" (UID: \"2fa901df-6086-43c3-8f7e-a7a633607486\") " pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-n4hgm" Oct 30 22:11:33.776557 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:33.776444 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fa901df-6086-43c3-8f7e-a7a633607486-credential-socket\") pod \"e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-n4hgm\" (UID: \"2fa901df-6086-43c3-8f7e-a7a633607486\") " pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-n4hgm" Oct 30 22:11:33.776557 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:33.776480 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fa901df-6086-43c3-8f7e-a7a633607486-istio-envoy\") pod \"e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-n4hgm\" (UID: \"2fa901df-6086-43c3-8f7e-a7a633607486\") " pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-n4hgm" Oct 30 22:11:33.776557 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:33.776548 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8hbn\" (UniqueName: \"kubernetes.io/projected/2fa901df-6086-43c3-8f7e-a7a633607486-kube-api-access-c8hbn\") pod \"e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-n4hgm\" (UID: \"2fa901df-6086-43c3-8f7e-a7a633607486\") " pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-n4hgm" Oct 30 22:11:33.776746 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:33.776630 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fa901df-6086-43c3-8f7e-a7a633607486-istio-token\") pod \"e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-n4hgm\" (UID: \"2fa901df-6086-43c3-8f7e-a7a633607486\") " pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-n4hgm" Oct 30 22:11:33.776746 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:33.776688 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fa901df-6086-43c3-8f7e-a7a633607486-istio-podinfo\") pod \"e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-n4hgm\" (UID: \"2fa901df-6086-43c3-8f7e-a7a633607486\") " pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-n4hgm" Oct 30 22:11:33.776746 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:33.776712 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fa901df-6086-43c3-8f7e-a7a633607486-istiod-ca-cert\") pod \"e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-n4hgm\" (UID: \"2fa901df-6086-43c3-8f7e-a7a633607486\") " pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-n4hgm" Oct 30 22:11:33.776892 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:33.776784 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fa901df-6086-43c3-8f7e-a7a633607486-istio-data\") pod \"e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-n4hgm\" (UID: \"2fa901df-6086-43c3-8f7e-a7a633607486\") " pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-n4hgm" Oct 30 22:11:33.776942 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:33.776923 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fa901df-6086-43c3-8f7e-a7a633607486-workload-certs\") pod \"e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-n4hgm\" (UID: \"2fa901df-6086-43c3-8f7e-a7a633607486\") " pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-n4hgm" Oct 30 22:11:33.878027 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:33.877988 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fa901df-6086-43c3-8f7e-a7a633607486-istio-token\") pod \"e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-n4hgm\" (UID: \"2fa901df-6086-43c3-8f7e-a7a633607486\") " pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-n4hgm" Oct 30 22:11:33.878229 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:33.878044 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fa901df-6086-43c3-8f7e-a7a633607486-istio-podinfo\") pod \"e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-n4hgm\" (UID: \"2fa901df-6086-43c3-8f7e-a7a633607486\") " pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-n4hgm" Oct 30 22:11:33.878229 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:33.878140 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fa901df-6086-43c3-8f7e-a7a633607486-istiod-ca-cert\") pod \"e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-n4hgm\" (UID: \"2fa901df-6086-43c3-8f7e-a7a633607486\") " pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-n4hgm" Oct 30 22:11:33.878229 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:33.878196 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fa901df-6086-43c3-8f7e-a7a633607486-istio-data\") pod \"e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-n4hgm\" (UID: \"2fa901df-6086-43c3-8f7e-a7a633607486\") " pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-n4hgm" Oct 30 22:11:33.878458 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:33.878274 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fa901df-6086-43c3-8f7e-a7a633607486-workload-certs\") pod \"e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-n4hgm\" (UID: \"2fa901df-6086-43c3-8f7e-a7a633607486\") " pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-n4hgm" Oct 30 22:11:33.878458 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:33.878330 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fa901df-6086-43c3-8f7e-a7a633607486-workload-socket\") pod \"e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-n4hgm\" (UID: \"2fa901df-6086-43c3-8f7e-a7a633607486\") " pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-n4hgm" Oct 30 22:11:33.878458 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:33.878444 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fa901df-6086-43c3-8f7e-a7a633607486-credential-socket\") pod \"e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-n4hgm\" (UID: \"2fa901df-6086-43c3-8f7e-a7a633607486\") " pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-n4hgm" Oct 30 22:11:33.878623 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:33.878473 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fa901df-6086-43c3-8f7e-a7a633607486-istio-envoy\") pod \"e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-n4hgm\" (UID: \"2fa901df-6086-43c3-8f7e-a7a633607486\") " pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-n4hgm" Oct 30 22:11:33.878623 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:33.878509 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8hbn\" (UniqueName: \"kubernetes.io/projected/2fa901df-6086-43c3-8f7e-a7a633607486-kube-api-access-c8hbn\") pod \"e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-n4hgm\" (UID: \"2fa901df-6086-43c3-8f7e-a7a633607486\") " pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-n4hgm" Oct 30 22:11:33.878970 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:33.878943 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fa901df-6086-43c3-8f7e-a7a633607486-workload-certs\") pod \"e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-n4hgm\" (UID: \"2fa901df-6086-43c3-8f7e-a7a633607486\") " pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-n4hgm" Oct 30 22:11:33.879116 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:33.878992 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fa901df-6086-43c3-8f7e-a7a633607486-credential-socket\") pod \"e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-n4hgm\" (UID: \"2fa901df-6086-43c3-8f7e-a7a633607486\") " pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-n4hgm" Oct 30 22:11:33.879116 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:33.879010 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fa901df-6086-43c3-8f7e-a7a633607486-istio-data\") pod \"e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-n4hgm\" (UID: \"2fa901df-6086-43c3-8f7e-a7a633607486\") " pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-n4hgm" Oct 30 22:11:33.879116 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:33.879068 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fa901df-6086-43c3-8f7e-a7a633607486-istiod-ca-cert\") pod \"e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-n4hgm\" (UID: \"2fa901df-6086-43c3-8f7e-a7a633607486\") " pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-n4hgm" Oct 30 22:11:33.879233 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:33.879037 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fa901df-6086-43c3-8f7e-a7a633607486-workload-socket\") pod \"e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-n4hgm\" (UID: \"2fa901df-6086-43c3-8f7e-a7a633607486\") " pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-n4hgm" Oct 30 22:11:33.880992 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:33.880955 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fa901df-6086-43c3-8f7e-a7a633607486-istio-podinfo\") pod \"e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-n4hgm\" (UID: \"2fa901df-6086-43c3-8f7e-a7a633607486\") " pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-n4hgm" Oct 30 22:11:33.881087 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:33.881004 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fa901df-6086-43c3-8f7e-a7a633607486-istio-envoy\") pod \"e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-n4hgm\" (UID: \"2fa901df-6086-43c3-8f7e-a7a633607486\") " pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-n4hgm" Oct 30 22:11:33.891252 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:33.891227 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fa901df-6086-43c3-8f7e-a7a633607486-istio-token\") pod \"e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-n4hgm\" (UID: \"2fa901df-6086-43c3-8f7e-a7a633607486\") " pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-n4hgm" Oct 30 22:11:33.891350 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:33.891318 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8hbn\" (UniqueName: \"kubernetes.io/projected/2fa901df-6086-43c3-8f7e-a7a633607486-kube-api-access-c8hbn\") pod \"e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-n4hgm\" (UID: \"2fa901df-6086-43c3-8f7e-a7a633607486\") " pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-n4hgm" Oct 30 22:11:33.946091 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:33.946065 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-5313f0-openshift-default-dockercfg-scpj8\"" Oct 30 22:11:33.952321 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:33.952293 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-n4hgm" Oct 30 22:11:34.248210 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:34.248179 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-n4hgm"] Oct 30 22:11:34.249997 ip-10-0-137-181 kubenswrapper[6720]: W1030 22:11:34.249965 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fa901df_6086_43c3_8f7e_a7a633607486.slice/crio-92015ddc3028564eb797fd02e1ba42a3b25f545a1fbab7df289575334f162528 WatchSource:0}: Error finding container 92015ddc3028564eb797fd02e1ba42a3b25f545a1fbab7df289575334f162528: Status 404 returned error can't find the container with id 92015ddc3028564eb797fd02e1ba42a3b25f545a1fbab7df289575334f162528 Oct 30 22:11:34.253834 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:34.253789 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:11:34.253949 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:34.253891 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:11:34.253949 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:34.253937 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:11:34.323591 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:34.323559 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-n4hgm" event={"ID":"2fa901df-6086-43c3-8f7e-a7a633607486","Type":"ContainerStarted","Data":"92015ddc3028564eb797fd02e1ba42a3b25f545a1fbab7df289575334f162528"} Oct 30 22:11:34.411426 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:34.411146 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-a5f192-585685f64d-2vwwn"] Oct 30 22:11:34.418409 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:34.417024 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a5f192-585685f64d-2vwwn" Oct 30 22:11:34.430728 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:34.428655 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-a5f192-serving-cert\"" Oct 30 22:11:34.439516 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:34.436301 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-a5f192-585685f64d-2vwwn"] Oct 30 22:11:34.520856 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:34.520773 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:34.588468 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:34.588433 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbxnq\" (UniqueName: \"kubernetes.io/projected/423a4a57-1e84-4833-ac24-261dcad52016-kube-api-access-dbxnq\") pod \"maas-api-e2e-extmod-b-a5f192-585685f64d-2vwwn\" (UID: \"423a4a57-1e84-4833-ac24-261dcad52016\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a5f192-585685f64d-2vwwn" Oct 30 22:11:34.588468 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:34.588484 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/423a4a57-1e84-4833-ac24-261dcad52016-metrics-certs\") pod \"maas-api-e2e-extmod-b-a5f192-585685f64d-2vwwn\" (UID: \"423a4a57-1e84-4833-ac24-261dcad52016\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a5f192-585685f64d-2vwwn" Oct 30 22:11:34.588922 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:34.588628 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/423a4a57-1e84-4833-ac24-261dcad52016-maas-api-tls\") pod \"maas-api-e2e-extmod-b-a5f192-585685f64d-2vwwn\" (UID: \"423a4a57-1e84-4833-ac24-261dcad52016\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a5f192-585685f64d-2vwwn" Oct 30 22:11:34.588922 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:34.588660 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/423a4a57-1e84-4833-ac24-261dcad52016-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-a5f192-585685f64d-2vwwn\" (UID: \"423a4a57-1e84-4833-ac24-261dcad52016\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a5f192-585685f64d-2vwwn" Oct 30 22:11:34.690085 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:34.690053 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/423a4a57-1e84-4833-ac24-261dcad52016-maas-api-tls\") pod \"maas-api-e2e-extmod-b-a5f192-585685f64d-2vwwn\" (UID: \"423a4a57-1e84-4833-ac24-261dcad52016\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a5f192-585685f64d-2vwwn" Oct 30 22:11:34.690085 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:34.690092 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/423a4a57-1e84-4833-ac24-261dcad52016-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-a5f192-585685f64d-2vwwn\" (UID: \"423a4a57-1e84-4833-ac24-261dcad52016\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a5f192-585685f64d-2vwwn" Oct 30 22:11:34.690530 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:34.690327 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbxnq\" (UniqueName: \"kubernetes.io/projected/423a4a57-1e84-4833-ac24-261dcad52016-kube-api-access-dbxnq\") pod \"maas-api-e2e-extmod-b-a5f192-585685f64d-2vwwn\" (UID: \"423a4a57-1e84-4833-ac24-261dcad52016\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a5f192-585685f64d-2vwwn" Oct 30 22:11:34.690530 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:34.690397 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/423a4a57-1e84-4833-ac24-261dcad52016-metrics-certs\") pod \"maas-api-e2e-extmod-b-a5f192-585685f64d-2vwwn\" (UID: \"423a4a57-1e84-4833-ac24-261dcad52016\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a5f192-585685f64d-2vwwn" Oct 30 22:11:34.690902 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:34.690880 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/423a4a57-1e84-4833-ac24-261dcad52016-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-a5f192-585685f64d-2vwwn\" (UID: \"423a4a57-1e84-4833-ac24-261dcad52016\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a5f192-585685f64d-2vwwn" Oct 30 22:11:34.693325 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:34.693303 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/423a4a57-1e84-4833-ac24-261dcad52016-metrics-certs\") pod \"maas-api-e2e-extmod-b-a5f192-585685f64d-2vwwn\" (UID: \"423a4a57-1e84-4833-ac24-261dcad52016\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a5f192-585685f64d-2vwwn" Oct 30 22:11:34.694785 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:34.694762 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/423a4a57-1e84-4833-ac24-261dcad52016-maas-api-tls\") pod \"maas-api-e2e-extmod-b-a5f192-585685f64d-2vwwn\" (UID: \"423a4a57-1e84-4833-ac24-261dcad52016\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a5f192-585685f64d-2vwwn" Oct 30 22:11:34.704802 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:34.704777 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbxnq\" (UniqueName: \"kubernetes.io/projected/423a4a57-1e84-4833-ac24-261dcad52016-kube-api-access-dbxnq\") pod \"maas-api-e2e-extmod-b-a5f192-585685f64d-2vwwn\" (UID: \"423a4a57-1e84-4833-ac24-261dcad52016\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a5f192-585685f64d-2vwwn" Oct 30 22:11:34.723282 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:34.723242 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:34.741922 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:34.741893 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a5f192-585685f64d-2vwwn" Oct 30 22:11:35.049753 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:35.049719 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-a5f192-585685f64d-2vwwn"] Oct 30 22:11:35.271693 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:35.270899 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:35.338721 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:35.338623 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-n4hgm" event={"ID":"2fa901df-6086-43c3-8f7e-a7a633607486","Type":"ContainerStarted","Data":"b98c418c3157438e619241611a2e22a614b3a8eb620d9a4f1227ba52f60b3b6b"} Oct 30 22:11:35.340544 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:35.340514 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a5f192-585685f64d-2vwwn" event={"ID":"423a4a57-1e84-4833-ac24-261dcad52016","Type":"ContainerStarted","Data":"fbfb890fa87340811035202c92760e4da9c2e8b473933ee509f28e07682e3739"} Oct 30 22:11:35.367262 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:35.367195 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-n4hgm" podStartSLOduration=2.367174375 podStartE2EDuration="2.367174375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:35.365934096 +0000 UTC m=+1861.536899304" watchObservedRunningTime="2026-10-30 22:11:35.367174375 +0000 UTC m=+1861.538139559" Oct 30 22:11:35.975465 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:35.975135 6720 patch_prober.go:29] interesting pod/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-n4hgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 22:11:35.975465 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:35.975203 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-n4hgm" podUID="2fa901df-6086-43c3-8f7e-a7a633607486" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 22:11:35.985473 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:35.984948 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-n4hgm" Oct 30 22:11:36.158448 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:36.156206 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:36.374436 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:36.374086 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a5f192-585685f64d-2vwwn" event={"ID":"423a4a57-1e84-4833-ac24-261dcad52016","Type":"ContainerStarted","Data":"e903569b3b6be233f8af498e382f7d34441979c029700e52fb4e5e76ecd71833"} Oct 30 22:11:36.374436 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:36.374177 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a5f192-585685f64d-2vwwn" Oct 30 22:11:36.398045 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:36.397994 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a5f192-585685f64d-2vwwn" podStartSLOduration=1.857068401 podStartE2EDuration="2.397981671s" totalImagesPullingTime="540.91327ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:36.395279057 +0000 UTC m=+1862.566244255" watchObservedRunningTime="2026-10-30 22:11:36.397981671 +0000 UTC m=+1862.568946871" Oct 30 22:11:36.957419 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:36.955781 6720 patch_prober.go:29] interesting pod/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-n4hgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 22:11:36.957419 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:36.955841 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-n4hgm" podUID="2fa901df-6086-43c3-8f7e-a7a633607486" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 22:11:36.983479 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:36.983284 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:37.963424 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:37.957864 6720 patch_prober.go:29] interesting pod/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-n4hgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 22:11:37.963424 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:37.957940 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-n4hgm" podUID="2fa901df-6086-43c3-8f7e-a7a633607486" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 22:11:38.096425 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:38.095866 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:38.969406 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:38.956813 6720 patch_prober.go:29] interesting pod/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-n4hgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 22:11:38.969406 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:38.956891 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-n4hgm" podUID="2fa901df-6086-43c3-8f7e-a7a633607486" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 22:11:39.311501 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:39.309920 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:39.970611 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:39.967986 6720 patch_prober.go:29] interesting pod/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-n4hgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 22:11:39.970611 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:39.968047 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-n4hgm" podUID="2fa901df-6086-43c3-8f7e-a7a633607486" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 22:11:40.045637 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:40.045600 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:40.417592 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:11:40.417547 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 22:11:40.953374 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:40.953337 6720 patch_prober.go:29] interesting pod/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-n4hgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 22:11:40.953674 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:40.953645 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-n4hgm" podUID="2fa901df-6086-43c3-8f7e-a7a633607486" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 22:11:41.098413 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:41.097360 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:41.269014 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:41.268912 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:41.964592 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:41.964556 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-n4hgm" Oct 30 22:11:41.965738 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:41.965710 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-n4hgm" Oct 30 22:11:41.966776 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:41.966747 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-n4hgm" Oct 30 22:11:42.023084 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:42.023022 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:42.393998 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:42.393967 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a5f192-585685f64d-2vwwn" Oct 30 22:11:42.496529 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:42.496496 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:42.645468 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:42.645286 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:42.807584 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:42.807552 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:43.064030 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:43.063935 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:43.233480 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:43.233439 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:43.387071 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:43.387035 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:43.557766 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:43.557721 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:43.819494 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:43.819412 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:43.983101 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:43.983059 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:44.219601 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:44.219567 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:44.732746 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:44.732677 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:44.888983 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:44.888898 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:45.612956 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:45.612913 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:45.927046 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:45.926912 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:46.415517 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:46.415478 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:47.250379 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:47.250341 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:48.013016 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:48.012981 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:48.710050 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:48.710009 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:49.430966 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:49.430933 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:50.239554 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:50.239513 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:51.005564 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:51.005530 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:51.806103 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:51.806070 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:52.415622 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:52.415522 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:53.140739 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:53.140699 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:53.825002 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:53.824952 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:54.576197 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:54.576150 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:55.416578 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:55.416536 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:56.112876 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:56.112829 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:56.966291 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:56.965479 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:57.637800 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:57.637757 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:58.227065 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:58.227025 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:58.423925 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:58.423894 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:58.933337 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:58.933298 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:59.719168 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:59.719120 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:11:59.901931 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:11:59.901895 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:00.090476 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:00.090354 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:00.265429 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:00.265374 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:00.432497 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:00.432459 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:00.575217 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:00.575162 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:00.993726 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:00.993689 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:01.157878 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:01.157827 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:01.665438 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:01.665404 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:01.832509 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:01.832474 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:02.068356 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:02.068227 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:02.740815 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:02.740780 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:03.511034 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:03.511000 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:04.348624 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:04.348590 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:05.052165 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:05.052130 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:05.846044 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:05.846004 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:06.535445 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:06.535346 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:07.221907 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:07.221863 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:07.949039 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:07.948995 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:08.721493 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:08.721453 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:09.485823 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:09.485784 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:10.141270 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:10.141237 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:10.403982 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:12:10.403897 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 22:12:10.784906 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:10.784797 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:11.580310 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:11.579026 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:12.231639 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:12.231611 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:12.415015 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:12.414958 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:12.904347 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:12.904288 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:13.568173 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:13.567948 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:14.229475 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:14.229376 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:14.945159 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:14.945101 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:15.377022 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:15.376987 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-f0f5e2-664fb64799-h7rnw"] Oct 30 22:12:15.377848 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:15.377791 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f0f5e2-664fb64799-h7rnw" podUID="2aa72ad3-ab65-436a-abd6-8bf4bda11bf3" containerName="maas-api" containerID="cri-o://d96a27a7e5b0aff876ebc829b91438ed6367ed10bf1c8c84e7a4c4a6da2431d1" gracePeriod=30 Oct 30 22:12:15.461579 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:15.461546 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:15.665457 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:15.659323 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:15.718872 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:15.718840 6720 generic.go:356] "Generic (PLEG): container finished" podID="2aa72ad3-ab65-436a-abd6-8bf4bda11bf3" containerID="d96a27a7e5b0aff876ebc829b91438ed6367ed10bf1c8c84e7a4c4a6da2431d1" exitCode=0 Oct 30 22:12:15.719137 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:15.719120 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f0f5e2-664fb64799-h7rnw" event={"ID":"2aa72ad3-ab65-436a-abd6-8bf4bda11bf3","Type":"ContainerDied","Data":"d96a27a7e5b0aff876ebc829b91438ed6367ed10bf1c8c84e7a4c4a6da2431d1"} Oct 30 22:12:15.842334 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:15.842292 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:15.945939 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:15.945912 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f0f5e2-664fb64799-h7rnw" Oct 30 22:12:15.989240 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:15.989213 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:16.046319 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:16.046284 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2aa72ad3-ab65-436a-abd6-8bf4bda11bf3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2aa72ad3-ab65-436a-abd6-8bf4bda11bf3-metrics-certs\") pod \"2aa72ad3-ab65-436a-abd6-8bf4bda11bf3\" (UID: \"2aa72ad3-ab65-436a-abd6-8bf4bda11bf3\") " Oct 30 22:12:16.046529 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:16.046361 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2aa72ad3-ab65-436a-abd6-8bf4bda11bf3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2aa72ad3-ab65-436a-abd6-8bf4bda11bf3-maas-api-tls\") pod \"2aa72ad3-ab65-436a-abd6-8bf4bda11bf3\" (UID: \"2aa72ad3-ab65-436a-abd6-8bf4bda11bf3\") " Oct 30 22:12:16.046529 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:16.046460 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2aa72ad3-ab65-436a-abd6-8bf4bda11bf3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2aa72ad3-ab65-436a-abd6-8bf4bda11bf3-openshift-service-ca\") pod \"2aa72ad3-ab65-436a-abd6-8bf4bda11bf3\" (UID: \"2aa72ad3-ab65-436a-abd6-8bf4bda11bf3\") " Oct 30 22:12:16.047264 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:16.046765 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2aa72ad3-ab65-436a-abd6-8bf4bda11bf3-kube-api-access-p9pxt\" (UniqueName: \"kubernetes.io/projected/2aa72ad3-ab65-436a-abd6-8bf4bda11bf3-kube-api-access-p9pxt\") pod \"2aa72ad3-ab65-436a-abd6-8bf4bda11bf3\" (UID: \"2aa72ad3-ab65-436a-abd6-8bf4bda11bf3\") " Oct 30 22:12:16.047264 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:16.046966 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2aa72ad3-ab65-436a-abd6-8bf4bda11bf3-openshift-service-ca" pod "2aa72ad3-ab65-436a-abd6-8bf4bda11bf3" (UID: "2aa72ad3-ab65-436a-abd6-8bf4bda11bf3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:12:16.047483 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:16.047377 6720 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2aa72ad3-ab65-436a-abd6-8bf4bda11bf3-openshift-service-ca\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:12:16.049095 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:16.049063 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2aa72ad3-ab65-436a-abd6-8bf4bda11bf3-metrics-certs" pod "2aa72ad3-ab65-436a-abd6-8bf4bda11bf3" (UID: "2aa72ad3-ab65-436a-abd6-8bf4bda11bf3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:12:16.049264 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:16.049092 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2aa72ad3-ab65-436a-abd6-8bf4bda11bf3-maas-api-tls" pod "2aa72ad3-ab65-436a-abd6-8bf4bda11bf3" (UID: "2aa72ad3-ab65-436a-abd6-8bf4bda11bf3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:12:16.050040 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:16.050018 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2aa72ad3-ab65-436a-abd6-8bf4bda11bf3-kube-api-access-p9pxt" pod "2aa72ad3-ab65-436a-abd6-8bf4bda11bf3" (UID: "2aa72ad3-ab65-436a-abd6-8bf4bda11bf3"). InnerVolumeSpecName "kube-api-access-p9pxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:12:16.149436 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:16.149376 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9pxt\" (UniqueName: \"kubernetes.io/projected/2aa72ad3-ab65-436a-abd6-8bf4bda11bf3-kube-api-access-p9pxt\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:12:16.149436 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:16.149437 6720 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2aa72ad3-ab65-436a-abd6-8bf4bda11bf3-metrics-certs\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:12:16.149729 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:16.149452 6720 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2aa72ad3-ab65-436a-abd6-8bf4bda11bf3-maas-api-tls\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:12:16.383602 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:16.383570 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:16.750427 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:16.750207 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f0f5e2-664fb64799-h7rnw" event={"ID":"2aa72ad3-ab65-436a-abd6-8bf4bda11bf3","Type":"ContainerDied","Data":"27377e72dfc0eb17f64af71e41499c6802cdbb6ac42359d457e899c98353771a"} Oct 30 22:12:16.750427 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:16.750284 6720 scope.go:122] "RemoveContainer" containerID="d96a27a7e5b0aff876ebc829b91438ed6367ed10bf1c8c84e7a4c4a6da2431d1" Oct 30 22:12:16.750722 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:16.750475 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-f0f5e2-664fb64799-h7rnw" Oct 30 22:12:16.798722 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:16.797892 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-f0f5e2-664fb64799-h7rnw"] Oct 30 22:12:16.814294 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:16.811563 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:16.814683 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:16.814644 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-f0f5e2-664fb64799-h7rnw"] Oct 30 22:12:17.624992 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:17.624943 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:17.967377 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:17.967276 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2aa72ad3-ab65-436a-abd6-8bf4bda11bf3" path="/var/lib/kubelet/pods/2aa72ad3-ab65-436a-abd6-8bf4bda11bf3/volumes" Oct 30 22:12:18.338500 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:18.330417 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:19.050705 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:19.050666 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:19.732502 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:19.730825 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:20.389360 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:20.389322 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:21.044059 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:21.044020 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:21.755673 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:21.755639 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:22.604638 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:22.604603 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:22.758967 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:22.758926 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:22.927283 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:22.927203 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:23.088411 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:23.084855 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:23.427987 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:23.427928 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:24.198734 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:24.198700 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:24.847698 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:24.847663 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:25.255293 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:25.255247 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:25.409912 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:25.409874 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:25.709066 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:25.709026 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:26.030054 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:26.029964 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:27.039602 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:27.039542 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:27.658021 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:27.657987 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:28.510634 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:28.510596 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:28.817349 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:28.817264 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:29.019982 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:29.019937 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:29.172931 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:29.172883 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:29.344455 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:29.344374 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:30.027666 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:30.027625 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:30.299969 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:30.299876 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:30.453142 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:30.453100 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:30.949595 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:30.949558 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:31.743356 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:31.741850 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:32.511213 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:32.511177 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:33.133626 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:33.133579 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:33.916903 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:33.916858 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:34.621073 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:34.621032 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:35.238343 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:35.238307 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:35.907339 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:35.907293 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:36.615264 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:36.615227 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:37.305101 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:37.305066 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:37.989754 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:37.989719 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:38.636117 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:38.636086 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:39.329902 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:39.329784 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:40.046632 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:40.046592 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:40.403655 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:12:40.403616 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 22:12:40.659302 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:40.659139 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:41.358090 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:41.356895 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:42.017950 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:42.017911 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:42.630761 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:42.630711 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:43.334667 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:43.334630 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:44.021018 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:44.020982 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:44.164265 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:44.164225 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:44.309451 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:44.309308 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:44.817467 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:44.817425 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:45.540444 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:45.540404 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:46.239167 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:46.239130 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:46.875119 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:46.875086 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:47.859344 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:47.859306 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:48.596409 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:48.594891 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:49.236677 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:49.236633 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:49.415242 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:49.415206 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:49.623133 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:49.623085 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:49.884279 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:49.884156 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:50.196496 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:50.191157 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:50.617519 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:50.617470 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:50.806934 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:50.806897 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:51.133802 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:51.133763 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:51.350303 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:51.350272 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:51.539978 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:51.539845 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:51.959869 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:51.959828 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:52.409314 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:52.409271 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:52.700751 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:52.700667 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:52.904425 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:52.900751 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:53.041225 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:53.041115 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:53.800998 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:53.800962 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:54.552163 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:54.552128 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:55.273514 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:55.273479 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:55.907304 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:55.907269 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:56.534104 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:56.534057 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:57.142661 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:57.142545 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:57.864569 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:57.864534 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:58.831900 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:58.831855 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:12:59.448351 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:12:59.448312 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:00.126797 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:00.126751 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:00.858418 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:00.858350 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:01.505691 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:01.504969 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:02.157909 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:02.157856 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:03.127905 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:03.127868 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:04.051437 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:04.051404 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:04.536713 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:04.536682 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:04.682923 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:04.682885 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:05.337761 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:05.337726 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:05.807927 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:05.807890 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:06.101923 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:06.101839 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:06.294473 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:06.294440 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:06.450728 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:06.450691 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:06.840161 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:06.840017 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:07.527009 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:07.526953 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:08.329932 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:08.329893 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:09.300038 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:09.299970 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:09.677677 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:09.677642 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:09.865125 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:09.865090 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:10.256756 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:10.256716 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:10.405346 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:13:10.405307 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 22:13:10.887778 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:10.887741 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:11.041549 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:11.041462 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:11.630710 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:11.630667 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:11.784424 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:11.782167 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:12.176153 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:12.176122 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:12.587916 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:12.587823 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:12.879412 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:12.874249 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:13.319353 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:13.319267 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:13.597880 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:13.597792 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:14.438532 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:14.438491 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:15.487265 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:15.487230 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:15.662408 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:15.661206 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:16.274248 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:16.274198 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:17.065217 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:17.065168 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:17.777564 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:17.777512 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:18.435600 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:18.435562 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:19.061187 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:19.061148 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:19.861816 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:19.861736 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:20.518038 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:20.517962 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:21.143780 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:21.143742 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:21.947706 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:21.947673 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:22.523183 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:22.519709 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:23.248761 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:23.248713 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:24.027650 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:24.027617 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:24.732273 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:24.732224 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:25.486754 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:25.486725 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:26.202204 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:26.202172 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:26.937729 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:26.937694 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:27.505887 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:27.505834 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-d8c835-848b64f74c-kn8h6"] Oct 30 22:13:27.507014 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:27.506990 6720 cpu_manager.go:414] "containerMap: removing container" podUID="2aa72ad3-ab65-436a-abd6-8bf4bda11bf3" containerName="maas-api" Oct 30 22:13:27.507200 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:27.507173 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2aa72ad3-ab65-436a-abd6-8bf4bda11bf3" containerName="maas-api" Oct 30 22:13:27.507354 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:27.507338 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="2aa72ad3-ab65-436a-abd6-8bf4bda11bf3" containerName="maas-api" Oct 30 22:13:27.511483 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:27.511461 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d8c835-848b64f74c-kn8h6" Oct 30 22:13:27.521018 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:27.520499 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-d8c835-serving-cert\"" Oct 30 22:13:27.526595 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:27.526572 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-d8c835-848b64f74c-kn8h6"] Oct 30 22:13:27.598481 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:27.598446 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:27.624023 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:27.623992 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w24xg\" (UniqueName: \"kubernetes.io/projected/ca63c13b-3454-470e-80a0-d59435e6533f-kube-api-access-w24xg\") pod \"maas-api-e2e-extmod-b-d8c835-848b64f74c-kn8h6\" (UID: \"ca63c13b-3454-470e-80a0-d59435e6533f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d8c835-848b64f74c-kn8h6" Oct 30 22:13:27.624173 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:27.624044 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca63c13b-3454-470e-80a0-d59435e6533f-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-d8c835-848b64f74c-kn8h6\" (UID: \"ca63c13b-3454-470e-80a0-d59435e6533f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d8c835-848b64f74c-kn8h6" Oct 30 22:13:27.624286 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:27.624270 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca63c13b-3454-470e-80a0-d59435e6533f-maas-api-tls\") pod \"maas-api-e2e-extmod-b-d8c835-848b64f74c-kn8h6\" (UID: \"ca63c13b-3454-470e-80a0-d59435e6533f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d8c835-848b64f74c-kn8h6" Oct 30 22:13:27.624323 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:27.624311 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca63c13b-3454-470e-80a0-d59435e6533f-metrics-certs\") pod \"maas-api-e2e-extmod-b-d8c835-848b64f74c-kn8h6\" (UID: \"ca63c13b-3454-470e-80a0-d59435e6533f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d8c835-848b64f74c-kn8h6" Oct 30 22:13:27.741977 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:27.741931 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca63c13b-3454-470e-80a0-d59435e6533f-maas-api-tls\") pod \"maas-api-e2e-extmod-b-d8c835-848b64f74c-kn8h6\" (UID: \"ca63c13b-3454-470e-80a0-d59435e6533f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d8c835-848b64f74c-kn8h6" Oct 30 22:13:27.743669 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:27.743640 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca63c13b-3454-470e-80a0-d59435e6533f-metrics-certs\") pod \"maas-api-e2e-extmod-b-d8c835-848b64f74c-kn8h6\" (UID: \"ca63c13b-3454-470e-80a0-d59435e6533f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d8c835-848b64f74c-kn8h6" Oct 30 22:13:27.744338 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:27.744275 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w24xg\" (UniqueName: \"kubernetes.io/projected/ca63c13b-3454-470e-80a0-d59435e6533f-kube-api-access-w24xg\") pod \"maas-api-e2e-extmod-b-d8c835-848b64f74c-kn8h6\" (UID: \"ca63c13b-3454-470e-80a0-d59435e6533f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d8c835-848b64f74c-kn8h6" Oct 30 22:13:27.744987 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:27.744967 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca63c13b-3454-470e-80a0-d59435e6533f-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-d8c835-848b64f74c-kn8h6\" (UID: \"ca63c13b-3454-470e-80a0-d59435e6533f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d8c835-848b64f74c-kn8h6" Oct 30 22:13:27.746313 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:27.746285 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca63c13b-3454-470e-80a0-d59435e6533f-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-d8c835-848b64f74c-kn8h6\" (UID: \"ca63c13b-3454-470e-80a0-d59435e6533f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d8c835-848b64f74c-kn8h6" Oct 30 22:13:27.751462 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:27.751312 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca63c13b-3454-470e-80a0-d59435e6533f-metrics-certs\") pod \"maas-api-e2e-extmod-b-d8c835-848b64f74c-kn8h6\" (UID: \"ca63c13b-3454-470e-80a0-d59435e6533f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d8c835-848b64f74c-kn8h6" Oct 30 22:13:27.751986 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:27.751954 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca63c13b-3454-470e-80a0-d59435e6533f-maas-api-tls\") pod \"maas-api-e2e-extmod-b-d8c835-848b64f74c-kn8h6\" (UID: \"ca63c13b-3454-470e-80a0-d59435e6533f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d8c835-848b64f74c-kn8h6" Oct 30 22:13:27.754519 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:27.753732 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:27.761852 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:27.761769 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w24xg\" (UniqueName: \"kubernetes.io/projected/ca63c13b-3454-470e-80a0-d59435e6533f-kube-api-access-w24xg\") pod \"maas-api-e2e-extmod-b-d8c835-848b64f74c-kn8h6\" (UID: \"ca63c13b-3454-470e-80a0-d59435e6533f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d8c835-848b64f74c-kn8h6" Oct 30 22:13:27.830300 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:27.830267 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d8c835-848b64f74c-kn8h6" Oct 30 22:13:28.118448 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:28.118415 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-d8c835-848b64f74c-kn8h6"] Oct 30 22:13:28.129704 ip-10-0-137-181 kubenswrapper[6720]: W1030 22:13:28.126074 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca63c13b_3454_470e_80a0_d59435e6533f.slice/crio-cfccf890722f9261ebc9a141eec6fbc3190dc7c696c3db4e1712ebc6af5c24ef WatchSource:0}: Error finding container cfccf890722f9261ebc9a141eec6fbc3190dc7c696c3db4e1712ebc6af5c24ef: Status 404 returned error can't find the container with id cfccf890722f9261ebc9a141eec6fbc3190dc7c696c3db4e1712ebc6af5c24ef Oct 30 22:13:28.337916 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:28.337877 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d8c835-848b64f74c-kn8h6" event={"ID":"ca63c13b-3454-470e-80a0-d59435e6533f","Type":"ContainerStarted","Data":"cfccf890722f9261ebc9a141eec6fbc3190dc7c696c3db4e1712ebc6af5c24ef"} Oct 30 22:13:28.634786 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:28.634266 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:28.862793 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:28.862752 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:29.333894 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:29.333794 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:29.352261 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:29.352207 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d8c835-848b64f74c-kn8h6" event={"ID":"ca63c13b-3454-470e-80a0-d59435e6533f","Type":"ContainerStarted","Data":"e93c564228de69eb6f83bd95c8127739133cf8f0e059e8e73f4e796ba48ad415"} Oct 30 22:13:29.353498 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:29.353469 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d8c835-848b64f74c-kn8h6" Oct 30 22:13:29.379750 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:29.379703 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d8c835-848b64f74c-kn8h6" podStartSLOduration=1.922765206 podStartE2EDuration="2.37968897s" totalImagesPullingTime="456.923764ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:29.379257056 +0000 UTC m=+1975.550222249" watchObservedRunningTime="2026-10-30 22:13:29.37968897 +0000 UTC m=+1975.550654151" Oct 30 22:13:30.095061 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:30.095021 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:30.848061 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:30.847982 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:31.444413 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:31.438776 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:32.233115 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:32.233079 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:32.924727 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:32.924690 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:33.530591 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:33.530552 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:34.286505 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:34.286471 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:34.950250 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:34.950216 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:35.612278 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:35.611075 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:36.296366 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:36.296320 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:36.374493 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:36.374464 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d8c835-848b64f74c-kn8h6" Oct 30 22:13:37.041146 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:37.041111 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:37.679661 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:37.678870 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:38.343140 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:38.343101 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:39.048861 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:39.048828 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:39.725251 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:39.725216 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:40.403878 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:13:40.403843 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 22:13:40.569606 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:40.569554 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:41.352929 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:41.352892 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:41.995918 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:41.995864 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:42.632045 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:42.632003 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:43.495316 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:43.495178 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:44.329516 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:44.329475 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:45.081752 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:45.080225 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:45.979842 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:45.979791 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:46.327283 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:46.327197 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:46.539313 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:46.539268 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:47.381742 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:47.381698 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:48.178232 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:48.178191 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:48.357317 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:48.357287 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:48.567699 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:48.567601 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:48.751404 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:48.751358 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:49.578005 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:49.577966 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:50.452030 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:50.451995 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:50.666914 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:50.666863 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:51.401579 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:51.401542 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:51.671565 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:51.671487 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:51.836797 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:51.836765 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:51.990758 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:51.989638 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:52.249338 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:52.249260 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:52.531314 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:52.531238 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:52.699226 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:52.699196 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:53.107111 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:53.107082 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:53.264630 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:53.264566 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:53.410252 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:53.410169 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:53.681085 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:53.680994 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:53.828057 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:53.828023 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:54.556912 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:54.556880 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:54.742933 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:54.742898 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:55.522859 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:55.522818 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:56.232550 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:56.232519 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:56.902214 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:56.902183 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:57.676596 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:57.676560 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:58.523306 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:58.523268 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:59.152645 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:59.152606 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:13:59.834851 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:13:59.834804 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:00.519557 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:00.519505 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:01.261641 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:01.261608 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:01.917471 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:01.917424 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:02.613280 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:02.613245 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:03.412185 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:03.412112 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:04.122645 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:04.122599 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:04.804408 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:04.799495 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:05.321664 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:05.321634 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:05.469999 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:05.469959 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:05.737613 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:05.737578 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:05.904811 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:05.904761 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:06.056259 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:06.056180 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:06.354068 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:06.354031 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:06.637907 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:06.637813 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:07.429096 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:07.429063 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:08.134268 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:08.134230 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:08.469368 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:08.469227 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:08.615133 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:08.615097 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:09.396410 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:09.396109 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:10.127846 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:10.127811 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:10.403854 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:14:10.403766 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 22:14:10.805955 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:10.805874 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:12.420554 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:12.420519 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:12.818506 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:12.818258 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:13.094350 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:13.094268 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:13.246669 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:13.246627 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:13.659702 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:13.659667 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:14.284789 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:14.284752 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:14.925766 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:14.925731 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:15.606812 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:15.606757 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:16.321463 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:16.321408 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:17.007784 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:17.007747 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:17.220424 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:17.216253 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:17.576135 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:17.576103 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:17.953509 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:17.953405 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:18.753099 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:18.753055 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:19.375012 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:19.374983 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:19.542329 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:19.542297 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:19.716272 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:19.716170 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:19.900172 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:19.900114 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:20.067644 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:20.067552 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:20.412797 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:20.412755 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:20.665714 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:20.665619 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:20.861477 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:20.861443 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:21.040328 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:21.040202 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:21.274179 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:21.274143 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:21.661860 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:21.661827 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:21.846248 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:21.846175 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:22.561140 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:22.561089 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:23.506626 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:23.506589 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:24.463068 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:24.463023 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:25.353667 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:25.353629 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:26.448493 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:26.448359 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:26.650213 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:26.650176 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:26.827462 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:26.827350 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:27.117412 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:27.110110 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:27.653269 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:27.653238 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:28.240981 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:28.240910 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:28.404363 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:28.404332 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:28.555643 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:28.555559 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:28.926512 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:28.926481 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:29.080265 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:29.080234 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:29.647053 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:29.647022 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:30.256512 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:30.256458 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:31.266618 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:31.266575 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:31.958251 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:31.958209 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:32.268760 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:32.268672 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:32.464946 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:32.464905 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:32.965644 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:32.965612 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:33.245997 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:33.245901 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:34.044411 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:34.041433 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:34.234051 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:34.234019 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:34.411039 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:34.411001 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:34.591268 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:34.591238 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:34.766334 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:34.766232 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:35.037092 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:35.036943 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:35.238843 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:35.238805 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:35.548199 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:35.548161 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:35.720451 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:35.720407 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:36.135874 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:36.135839 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:36.325095 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:36.325061 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:36.481160 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:36.481068 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:37.125593 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:37.125554 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:37.335269 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:37.335237 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:37.492491 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:37.492404 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:37.732302 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:37.732269 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:37.987692 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:37.987630 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:38.254469 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:38.251986 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:38.419604 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:38.419552 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:39.012647 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:39.012617 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:39.169343 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:39.169306 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:39.476998 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:39.476964 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:39.750202 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:39.750131 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:39.940590 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:39.940552 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:40.409206 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:14:40.409166 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 22:14:40.798194 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:40.798092 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:41.481527 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:41.481488 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:41.708308 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:41.708277 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:42.411691 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:42.411592 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:43.117708 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:43.117640 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:43.288778 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:43.288748 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:43.964936 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:43.964906 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:44.771066 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:44.771028 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:45.614820 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:45.614785 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:46.359168 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:46.359084 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:47.180888 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:47.180792 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:47.415286 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:47.415243 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:47.849116 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:47.849068 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:48.145281 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:48.145234 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:48.341738 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:48.341705 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:48.661241 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:48.661205 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:48.835876 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:48.835830 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:49.824653 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:49.824602 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:50.400814 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:50.400779 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-l6qmw"] Oct 30 22:14:50.406207 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:50.406180 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-l6qmw" Oct 30 22:14:50.411271 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:50.411247 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-28b4bb-openshift-default-dockercfg-bdj78\"" Oct 30 22:14:50.423343 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:50.423317 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-l6qmw"] Oct 30 22:14:50.511934 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:50.511898 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad5f36c9-9a30-43b1-834b-3ab9c458bdb0-istio-envoy\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-l6qmw\" (UID: \"ad5f36c9-9a30-43b1-834b-3ab9c458bdb0\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-l6qmw" Oct 30 22:14:50.512111 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:50.512009 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad5f36c9-9a30-43b1-834b-3ab9c458bdb0-istio-podinfo\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-l6qmw\" (UID: \"ad5f36c9-9a30-43b1-834b-3ab9c458bdb0\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-l6qmw" Oct 30 22:14:50.512111 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:50.512072 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad5f36c9-9a30-43b1-834b-3ab9c458bdb0-credential-socket\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-l6qmw\" (UID: \"ad5f36c9-9a30-43b1-834b-3ab9c458bdb0\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-l6qmw" Oct 30 22:14:50.512197 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:50.512174 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad5f36c9-9a30-43b1-834b-3ab9c458bdb0-istio-data\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-l6qmw\" (UID: \"ad5f36c9-9a30-43b1-834b-3ab9c458bdb0\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-l6qmw" Oct 30 22:14:50.512232 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:50.512196 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad5f36c9-9a30-43b1-834b-3ab9c458bdb0-workload-certs\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-l6qmw\" (UID: \"ad5f36c9-9a30-43b1-834b-3ab9c458bdb0\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-l6qmw" Oct 30 22:14:50.512232 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:50.512218 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad5f36c9-9a30-43b1-834b-3ab9c458bdb0-istiod-ca-cert\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-l6qmw\" (UID: \"ad5f36c9-9a30-43b1-834b-3ab9c458bdb0\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-l6qmw" Oct 30 22:14:50.512314 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:50.512242 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-977js\" (UniqueName: \"kubernetes.io/projected/ad5f36c9-9a30-43b1-834b-3ab9c458bdb0-kube-api-access-977js\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-l6qmw\" (UID: \"ad5f36c9-9a30-43b1-834b-3ab9c458bdb0\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-l6qmw" Oct 30 22:14:50.512314 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:50.512305 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad5f36c9-9a30-43b1-834b-3ab9c458bdb0-istio-token\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-l6qmw\" (UID: \"ad5f36c9-9a30-43b1-834b-3ab9c458bdb0\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-l6qmw" Oct 30 22:14:50.512418 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:50.512379 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad5f36c9-9a30-43b1-834b-3ab9c458bdb0-workload-socket\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-l6qmw\" (UID: \"ad5f36c9-9a30-43b1-834b-3ab9c458bdb0\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-l6qmw" Oct 30 22:14:50.613721 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:50.613687 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad5f36c9-9a30-43b1-834b-3ab9c458bdb0-istio-data\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-l6qmw\" (UID: \"ad5f36c9-9a30-43b1-834b-3ab9c458bdb0\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-l6qmw" Oct 30 22:14:50.613899 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:50.613731 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad5f36c9-9a30-43b1-834b-3ab9c458bdb0-workload-certs\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-l6qmw\" (UID: \"ad5f36c9-9a30-43b1-834b-3ab9c458bdb0\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-l6qmw" Oct 30 22:14:50.613899 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:50.613851 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad5f36c9-9a30-43b1-834b-3ab9c458bdb0-istiod-ca-cert\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-l6qmw\" (UID: \"ad5f36c9-9a30-43b1-834b-3ab9c458bdb0\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-l6qmw" Oct 30 22:14:50.613899 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:50.613896 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-977js\" (UniqueName: \"kubernetes.io/projected/ad5f36c9-9a30-43b1-834b-3ab9c458bdb0-kube-api-access-977js\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-l6qmw\" (UID: \"ad5f36c9-9a30-43b1-834b-3ab9c458bdb0\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-l6qmw" Oct 30 22:14:50.614105 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:50.614033 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad5f36c9-9a30-43b1-834b-3ab9c458bdb0-istio-token\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-l6qmw\" (UID: \"ad5f36c9-9a30-43b1-834b-3ab9c458bdb0\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-l6qmw" Oct 30 22:14:50.614105 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:50.614084 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad5f36c9-9a30-43b1-834b-3ab9c458bdb0-istio-data\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-l6qmw\" (UID: \"ad5f36c9-9a30-43b1-834b-3ab9c458bdb0\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-l6qmw" Oct 30 22:14:50.614222 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:50.614110 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad5f36c9-9a30-43b1-834b-3ab9c458bdb0-workload-socket\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-l6qmw\" (UID: \"ad5f36c9-9a30-43b1-834b-3ab9c458bdb0\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-l6qmw" Oct 30 22:14:50.614222 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:50.614116 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad5f36c9-9a30-43b1-834b-3ab9c458bdb0-workload-certs\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-l6qmw\" (UID: \"ad5f36c9-9a30-43b1-834b-3ab9c458bdb0\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-l6qmw" Oct 30 22:14:50.614401 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:50.614237 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad5f36c9-9a30-43b1-834b-3ab9c458bdb0-istio-envoy\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-l6qmw\" (UID: \"ad5f36c9-9a30-43b1-834b-3ab9c458bdb0\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-l6qmw" Oct 30 22:14:50.614401 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:50.614302 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad5f36c9-9a30-43b1-834b-3ab9c458bdb0-istio-podinfo\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-l6qmw\" (UID: \"ad5f36c9-9a30-43b1-834b-3ab9c458bdb0\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-l6qmw" Oct 30 22:14:50.614401 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:50.614325 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad5f36c9-9a30-43b1-834b-3ab9c458bdb0-workload-socket\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-l6qmw\" (UID: \"ad5f36c9-9a30-43b1-834b-3ab9c458bdb0\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-l6qmw" Oct 30 22:14:50.614401 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:50.614368 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad5f36c9-9a30-43b1-834b-3ab9c458bdb0-credential-socket\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-l6qmw\" (UID: \"ad5f36c9-9a30-43b1-834b-3ab9c458bdb0\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-l6qmw" Oct 30 22:14:50.614676 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:50.614657 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad5f36c9-9a30-43b1-834b-3ab9c458bdb0-credential-socket\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-l6qmw\" (UID: \"ad5f36c9-9a30-43b1-834b-3ab9c458bdb0\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-l6qmw" Oct 30 22:14:50.614739 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:50.614704 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad5f36c9-9a30-43b1-834b-3ab9c458bdb0-istiod-ca-cert\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-l6qmw\" (UID: \"ad5f36c9-9a30-43b1-834b-3ab9c458bdb0\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-l6qmw" Oct 30 22:14:50.616684 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:50.616662 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad5f36c9-9a30-43b1-834b-3ab9c458bdb0-istio-envoy\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-l6qmw\" (UID: \"ad5f36c9-9a30-43b1-834b-3ab9c458bdb0\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-l6qmw" Oct 30 22:14:50.616907 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:50.616890 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad5f36c9-9a30-43b1-834b-3ab9c458bdb0-istio-podinfo\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-l6qmw\" (UID: \"ad5f36c9-9a30-43b1-834b-3ab9c458bdb0\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-l6qmw" Oct 30 22:14:50.626511 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:50.626487 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad5f36c9-9a30-43b1-834b-3ab9c458bdb0-istio-token\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-l6qmw\" (UID: \"ad5f36c9-9a30-43b1-834b-3ab9c458bdb0\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-l6qmw" Oct 30 22:14:50.626620 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:50.626549 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-977js\" (UniqueName: \"kubernetes.io/projected/ad5f36c9-9a30-43b1-834b-3ab9c458bdb0-kube-api-access-977js\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-l6qmw\" (UID: \"ad5f36c9-9a30-43b1-834b-3ab9c458bdb0\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-l6qmw" Oct 30 22:14:50.723744 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:50.723664 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-l6qmw" Oct 30 22:14:50.747838 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:50.747803 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:50.901145 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:50.901113 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:51.018366 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:51.018325 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-l6qmw"] Oct 30 22:14:51.019674 ip-10-0-137-181 kubenswrapper[6720]: W1030 22:14:51.019648 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad5f36c9_9a30_43b1_834b_3ab9c458bdb0.slice/crio-11b2aa5a1383bac58135b87cf09e69a1cfd9d50ba6ae6aa0730ade8460ca05b4 WatchSource:0}: Error finding container 11b2aa5a1383bac58135b87cf09e69a1cfd9d50ba6ae6aa0730ade8460ca05b4: Status 404 returned error can't find the container with id 11b2aa5a1383bac58135b87cf09e69a1cfd9d50ba6ae6aa0730ade8460ca05b4 Oct 30 22:14:51.022681 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:51.022647 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:14:51.022808 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:51.022730 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:14:51.022808 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:51.022786 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:14:51.547316 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:51.547268 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:52.015126 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:52.015091 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-l6qmw" event={"ID":"ad5f36c9-9a30-43b1-834b-3ab9c458bdb0","Type":"ContainerStarted","Data":"95c8c2ee943f94b7c8753a117968f13599665423a367e74b2377d2b3c9127829"} Oct 30 22:14:52.015700 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:52.015165 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-l6qmw" event={"ID":"ad5f36c9-9a30-43b1-834b-3ab9c458bdb0","Type":"ContainerStarted","Data":"11b2aa5a1383bac58135b87cf09e69a1cfd9d50ba6ae6aa0730ade8460ca05b4"} Oct 30 22:14:52.040929 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:52.040878 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-l6qmw" podStartSLOduration=2.040864941 podStartE2EDuration="2.040864941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:14:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:14:52.040744795 +0000 UTC m=+2058.211709976" watchObservedRunningTime="2026-10-30 22:14:52.040864941 +0000 UTC m=+2058.211830123" Oct 30 22:14:52.331272 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:52.331169 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:52.724982 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:52.724946 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-l6qmw" Oct 30 22:14:52.725610 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:52.725588 6720 patch_prober.go:29] interesting pod/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-l6qmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Oct 30 22:14:52.725668 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:52.725636 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-l6qmw" podUID="ad5f36c9-9a30-43b1-834b-3ab9c458bdb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Oct 30 22:14:53.216688 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:53.216658 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:53.515339 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:53.515249 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:53.686090 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:53.686056 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:53.728865 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:53.728832 6720 patch_prober.go:29] interesting pod/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-l6qmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Oct 30 22:14:53.730119 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:53.730088 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-l6qmw" podUID="ad5f36c9-9a30-43b1-834b-3ab9c458bdb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Oct 30 22:14:54.630906 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:54.630871 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:54.727937 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:54.727903 6720 patch_prober.go:29] interesting pod/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-l6qmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Oct 30 22:14:54.728621 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:54.728590 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-l6qmw" podUID="ad5f36c9-9a30-43b1-834b-3ab9c458bdb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Oct 30 22:14:55.126324 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:55.126284 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:55.542565 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:55.533928 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:55.724782 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:55.724744 6720 patch_prober.go:29] interesting pod/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-l6qmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Oct 30 22:14:55.725247 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:55.724799 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-l6qmw" podUID="ad5f36c9-9a30-43b1-834b-3ab9c458bdb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Oct 30 22:14:56.327432 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:56.327400 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:56.724817 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:56.724777 6720 patch_prober.go:29] interesting pod/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-l6qmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body= Oct 30 22:14:56.725254 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:56.724847 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-l6qmw" podUID="ad5f36c9-9a30-43b1-834b-3ab9c458bdb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" Oct 30 22:14:56.805579 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:56.805527 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:56.976368 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:56.976278 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:57.160358 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:57.160322 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:57.852552 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:57.852512 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:57.900308 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:57.900245 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-l6qmw" Oct 30 22:14:57.901240 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:57.901217 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-l6qmw" Oct 30 22:14:57.902338 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:57.902312 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-l6qmw" Oct 30 22:14:58.684323 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:58.684287 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:14:59.410012 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:14:59.409979 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:00.055098 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:00.055034 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:00.395886 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:00.395837 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6f8222-29889975-zb59n"] Oct 30 22:15:00.400773 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:00.400727 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6f8222-29889975-zb59n" Oct 30 22:15:00.438525 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:00.438497 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6f8222-29889975-zb59n"] Oct 30 22:15:00.444735 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:00.444316 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ggpf5\"" Oct 30 22:15:00.456334 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:00.456297 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ded6d4c9-cc20-4da2-bcb5-02f07193b223-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-6f8222-29889975-zb59n\" (UID: \"ded6d4c9-cc20-4da2-bcb5-02f07193b223\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6f8222-29889975-zb59n" Oct 30 22:15:00.456836 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:00.456421 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxzbq\" (UniqueName: \"kubernetes.io/projected/ded6d4c9-cc20-4da2-bcb5-02f07193b223-kube-api-access-mxzbq\") pod \"maas-api-key-cleanup-e2e-extmod-a-6f8222-29889975-zb59n\" (UID: \"ded6d4c9-cc20-4da2-bcb5-02f07193b223\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6f8222-29889975-zb59n" Oct 30 22:15:00.461163 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:00.461136 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0f73c9-29889975-bpmgx"] Oct 30 22:15:00.465726 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:00.465706 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0f73c9-29889975-bpmgx" Oct 30 22:15:00.491480 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:00.491443 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bacea0-29889975-gpxpm"] Oct 30 22:15:00.519741 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:00.519700 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0f73c9-29889975-bpmgx"] Oct 30 22:15:00.519741 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:00.519739 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bacea0-29889975-gpxpm"] Oct 30 22:15:00.519974 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:00.519811 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bacea0-29889975-gpxpm" Oct 30 22:15:00.558660 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:00.558488 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ded6d4c9-cc20-4da2-bcb5-02f07193b223-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-6f8222-29889975-zb59n\" (UID: \"ded6d4c9-cc20-4da2-bcb5-02f07193b223\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6f8222-29889975-zb59n" Oct 30 22:15:00.558660 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:00.558560 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxzbq\" (UniqueName: \"kubernetes.io/projected/ded6d4c9-cc20-4da2-bcb5-02f07193b223-kube-api-access-mxzbq\") pod \"maas-api-key-cleanup-e2e-extmod-a-6f8222-29889975-zb59n\" (UID: \"ded6d4c9-cc20-4da2-bcb5-02f07193b223\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6f8222-29889975-zb59n" Oct 30 22:15:00.558660 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:00.558603 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6b2208e-5fa5-4f8e-9fd3-c12ce229b361-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-0f73c9-29889975-bpmgx\" (UID: \"d6b2208e-5fa5-4f8e-9fd3-c12ce229b361\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0f73c9-29889975-bpmgx" Oct 30 22:15:00.558953 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:00.558729 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mwgf\" (UniqueName: \"kubernetes.io/projected/d6b2208e-5fa5-4f8e-9fd3-c12ce229b361-kube-api-access-9mwgf\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-0f73c9-29889975-bpmgx\" (UID: \"d6b2208e-5fa5-4f8e-9fd3-c12ce229b361\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0f73c9-29889975-bpmgx" Oct 30 22:15:00.559276 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:00.559255 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ded6d4c9-cc20-4da2-bcb5-02f07193b223-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-6f8222-29889975-zb59n\" (UID: \"ded6d4c9-cc20-4da2-bcb5-02f07193b223\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6f8222-29889975-zb59n" Oct 30 22:15:00.575762 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:00.575737 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxzbq\" (UniqueName: \"kubernetes.io/projected/ded6d4c9-cc20-4da2-bcb5-02f07193b223-kube-api-access-mxzbq\") pod \"maas-api-key-cleanup-e2e-extmod-a-6f8222-29889975-zb59n\" (UID: \"ded6d4c9-cc20-4da2-bcb5-02f07193b223\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6f8222-29889975-zb59n" Oct 30 22:15:00.660735 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:00.660644 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mwgf\" (UniqueName: \"kubernetes.io/projected/d6b2208e-5fa5-4f8e-9fd3-c12ce229b361-kube-api-access-9mwgf\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-0f73c9-29889975-bpmgx\" (UID: \"d6b2208e-5fa5-4f8e-9fd3-c12ce229b361\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0f73c9-29889975-bpmgx" Oct 30 22:15:00.660735 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:00.660688 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx676\" (UniqueName: \"kubernetes.io/projected/f1c46564-6708-4ed7-912e-9adfce46553e-kube-api-access-sx676\") pod \"maas-api-key-cleanup-e2e-extmod-a-bacea0-29889975-gpxpm\" (UID: \"f1c46564-6708-4ed7-912e-9adfce46553e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bacea0-29889975-gpxpm" Oct 30 22:15:00.660958 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:00.660934 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6b2208e-5fa5-4f8e-9fd3-c12ce229b361-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-0f73c9-29889975-bpmgx\" (UID: \"d6b2208e-5fa5-4f8e-9fd3-c12ce229b361\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0f73c9-29889975-bpmgx" Oct 30 22:15:00.661011 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:00.660999 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1c46564-6708-4ed7-912e-9adfce46553e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-bacea0-29889975-gpxpm\" (UID: \"f1c46564-6708-4ed7-912e-9adfce46553e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bacea0-29889975-gpxpm" Oct 30 22:15:00.661649 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:00.661631 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6b2208e-5fa5-4f8e-9fd3-c12ce229b361-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-0f73c9-29889975-bpmgx\" (UID: \"d6b2208e-5fa5-4f8e-9fd3-c12ce229b361\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0f73c9-29889975-bpmgx" Oct 30 22:15:00.676775 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:00.676748 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mwgf\" (UniqueName: \"kubernetes.io/projected/d6b2208e-5fa5-4f8e-9fd3-c12ce229b361-kube-api-access-9mwgf\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-0f73c9-29889975-bpmgx\" (UID: \"d6b2208e-5fa5-4f8e-9fd3-c12ce229b361\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0f73c9-29889975-bpmgx" Oct 30 22:15:00.717923 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:00.717873 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6f8222-29889975-zb59n" Oct 30 22:15:00.761931 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:00.761896 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1c46564-6708-4ed7-912e-9adfce46553e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-bacea0-29889975-gpxpm\" (UID: \"f1c46564-6708-4ed7-912e-9adfce46553e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bacea0-29889975-gpxpm" Oct 30 22:15:00.762111 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:00.761997 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx676\" (UniqueName: \"kubernetes.io/projected/f1c46564-6708-4ed7-912e-9adfce46553e-kube-api-access-sx676\") pod \"maas-api-key-cleanup-e2e-extmod-a-bacea0-29889975-gpxpm\" (UID: \"f1c46564-6708-4ed7-912e-9adfce46553e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bacea0-29889975-gpxpm" Oct 30 22:15:00.762696 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:00.762672 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1c46564-6708-4ed7-912e-9adfce46553e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-bacea0-29889975-gpxpm\" (UID: \"f1c46564-6708-4ed7-912e-9adfce46553e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bacea0-29889975-gpxpm" Oct 30 22:15:00.786753 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:00.786700 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx676\" (UniqueName: \"kubernetes.io/projected/f1c46564-6708-4ed7-912e-9adfce46553e-kube-api-access-sx676\") pod \"maas-api-key-cleanup-e2e-extmod-a-bacea0-29889975-gpxpm\" (UID: \"f1c46564-6708-4ed7-912e-9adfce46553e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bacea0-29889975-gpxpm" Oct 30 22:15:00.796604 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:00.794996 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0f73c9-29889975-bpmgx" Oct 30 22:15:00.836218 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:00.836193 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bacea0-29889975-gpxpm" Oct 30 22:15:01.024774 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:01.024748 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6f8222-29889975-zb59n"] Oct 30 22:15:01.026141 ip-10-0-137-181 kubenswrapper[6720]: W1030 22:15:01.026110 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podded6d4c9_cc20_4da2_bcb5_02f07193b223.slice/crio-97871b31bf5ce62d15bac8a397e9646c74f5ef6bd101484e83db966e2d4e3a69 WatchSource:0}: Error finding container 97871b31bf5ce62d15bac8a397e9646c74f5ef6bd101484e83db966e2d4e3a69: Status 404 returned error can't find the container with id 97871b31bf5ce62d15bac8a397e9646c74f5ef6bd101484e83db966e2d4e3a69 Oct 30 22:15:01.120956 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:01.120913 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6f8222-29889975-zb59n" event={"ID":"ded6d4c9-cc20-4da2-bcb5-02f07193b223","Type":"ContainerStarted","Data":"97871b31bf5ce62d15bac8a397e9646c74f5ef6bd101484e83db966e2d4e3a69"} Oct 30 22:15:01.126311 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:01.126271 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0f73c9-29889975-bpmgx"] Oct 30 22:15:01.127512 ip-10-0-137-181 kubenswrapper[6720]: W1030 22:15:01.127485 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6b2208e_5fa5_4f8e_9fd3_c12ce229b361.slice/crio-7eac2058be8dba15ef5aa2a804fb1221feb36ae01e239669f0d196b33718231f WatchSource:0}: Error finding container 7eac2058be8dba15ef5aa2a804fb1221feb36ae01e239669f0d196b33718231f: Status 404 returned error can't find the container with id 7eac2058be8dba15ef5aa2a804fb1221feb36ae01e239669f0d196b33718231f Oct 30 22:15:01.175406 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:01.175330 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bacea0-29889975-gpxpm"] Oct 30 22:15:01.176480 ip-10-0-137-181 kubenswrapper[6720]: W1030 22:15:01.176458 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1c46564_6708_4ed7_912e_9adfce46553e.slice/crio-86ad8bc172ac7bb5a2fc4308b45a99ebd4dabb1440374b72579d20950695a19d WatchSource:0}: Error finding container 86ad8bc172ac7bb5a2fc4308b45a99ebd4dabb1440374b72579d20950695a19d: Status 404 returned error can't find the container with id 86ad8bc172ac7bb5a2fc4308b45a99ebd4dabb1440374b72579d20950695a19d Oct 30 22:15:01.568318 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:01.568289 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:02.131711 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:02.131668 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6f8222-29889975-zb59n" event={"ID":"ded6d4c9-cc20-4da2-bcb5-02f07193b223","Type":"ContainerStarted","Data":"6c8795cca96f3c2a3f6fbf6c77fd15f7ec92dc970d125831369f5a4d0c33804c"} Oct 30 22:15:02.173975 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:02.173532 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6f8222-29889975-zb59n" podStartSLOduration=2.173511537 podStartE2EDuration="2.173511537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:02.171144339 +0000 UTC m=+2068.342109521" watchObservedRunningTime="2026-10-30 22:15:02.173511537 +0000 UTC m=+2068.344476723" Oct 30 22:15:02.178047 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:02.177589 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bacea0-29889975-gpxpm" event={"ID":"f1c46564-6708-4ed7-912e-9adfce46553e","Type":"ContainerStarted","Data":"9961e00ac751bf83b5a556ed697a4faf37cc124ea69b5d1b0433ec16232b3899"} Oct 30 22:15:02.178047 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:02.177672 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bacea0-29889975-gpxpm" event={"ID":"f1c46564-6708-4ed7-912e-9adfce46553e","Type":"ContainerStarted","Data":"86ad8bc172ac7bb5a2fc4308b45a99ebd4dabb1440374b72579d20950695a19d"} Oct 30 22:15:02.186949 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:02.186242 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0f73c9-29889975-bpmgx" event={"ID":"d6b2208e-5fa5-4f8e-9fd3-c12ce229b361","Type":"ContainerStarted","Data":"74032b3e7bb51482c85a58c6a32fc938438e4985a7c5b89d8cd7838fa77fb4f9"} Oct 30 22:15:02.186949 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:02.186291 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0f73c9-29889975-bpmgx" event={"ID":"d6b2208e-5fa5-4f8e-9fd3-c12ce229b361","Type":"ContainerStarted","Data":"7eac2058be8dba15ef5aa2a804fb1221feb36ae01e239669f0d196b33718231f"} Oct 30 22:15:02.245109 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:02.245031 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bacea0-29889975-gpxpm" podStartSLOduration=2.244993798 podStartE2EDuration="2.244993798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:02.218472821 +0000 UTC m=+2068.389438003" watchObservedRunningTime="2026-10-30 22:15:02.244993798 +0000 UTC m=+2068.415958981" Oct 30 22:15:02.909715 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:02.909642 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:03.196475 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:03.196369 6720 generic.go:356] "Generic (PLEG): container finished" podID="d6b2208e-5fa5-4f8e-9fd3-c12ce229b361" containerID="74032b3e7bb51482c85a58c6a32fc938438e4985a7c5b89d8cd7838fa77fb4f9" exitCode=60 Oct 30 22:15:03.196475 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:03.196446 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0f73c9-29889975-bpmgx" event={"ID":"d6b2208e-5fa5-4f8e-9fd3-c12ce229b361","Type":"ContainerDied","Data":"74032b3e7bb51482c85a58c6a32fc938438e4985a7c5b89d8cd7838fa77fb4f9"} Oct 30 22:15:03.196848 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:03.196833 6720 scope.go:122] "RemoveContainer" containerID="74032b3e7bb51482c85a58c6a32fc938438e4985a7c5b89d8cd7838fa77fb4f9" Oct 30 22:15:03.198472 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:03.198451 6720 generic.go:356] "Generic (PLEG): container finished" podID="ded6d4c9-cc20-4da2-bcb5-02f07193b223" containerID="6c8795cca96f3c2a3f6fbf6c77fd15f7ec92dc970d125831369f5a4d0c33804c" exitCode=60 Oct 30 22:15:03.198601 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:03.198520 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6f8222-29889975-zb59n" event={"ID":"ded6d4c9-cc20-4da2-bcb5-02f07193b223","Type":"ContainerDied","Data":"6c8795cca96f3c2a3f6fbf6c77fd15f7ec92dc970d125831369f5a4d0c33804c"} Oct 30 22:15:03.198797 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:03.198778 6720 scope.go:122] "RemoveContainer" containerID="6c8795cca96f3c2a3f6fbf6c77fd15f7ec92dc970d125831369f5a4d0c33804c" Oct 30 22:15:03.200503 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:03.200462 6720 generic.go:356] "Generic (PLEG): container finished" podID="f1c46564-6708-4ed7-912e-9adfce46553e" containerID="9961e00ac751bf83b5a556ed697a4faf37cc124ea69b5d1b0433ec16232b3899" exitCode=60 Oct 30 22:15:03.200611 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:03.200564 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bacea0-29889975-gpxpm" event={"ID":"f1c46564-6708-4ed7-912e-9adfce46553e","Type":"ContainerDied","Data":"9961e00ac751bf83b5a556ed697a4faf37cc124ea69b5d1b0433ec16232b3899"} Oct 30 22:15:03.200817 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:03.200795 6720 scope.go:122] "RemoveContainer" containerID="9961e00ac751bf83b5a556ed697a4faf37cc124ea69b5d1b0433ec16232b3899" Oct 30 22:15:04.059998 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:04.059969 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:04.230992 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:04.230897 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0f73c9-29889975-bpmgx" event={"ID":"d6b2208e-5fa5-4f8e-9fd3-c12ce229b361","Type":"ContainerStarted","Data":"7a929bcb4efc1d93eeca49ace8d07a33992b18d19b64ead7dd9635bf4a97f8b2"} Oct 30 22:15:04.234084 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:04.234041 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6f8222-29889975-zb59n" event={"ID":"ded6d4c9-cc20-4da2-bcb5-02f07193b223","Type":"ContainerStarted","Data":"37826089595cb5ca5b5bf101d4c58d5115a6fa35d9038e7f585bd89e2c77d704"} Oct 30 22:15:04.237362 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:04.237336 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bacea0-29889975-gpxpm" event={"ID":"f1c46564-6708-4ed7-912e-9adfce46553e","Type":"ContainerStarted","Data":"c79393a5740a881edbd0e232448e4479036f0cea02ed0178b76d495bbba7a317"} Oct 30 22:15:04.260033 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:04.258984 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0f73c9-29889975-bpmgx" podStartSLOduration=4.258964945 podStartE2EDuration="4.258964945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:02.272718593 +0000 UTC m=+2068.443683777" watchObservedRunningTime="2026-10-30 22:15:04.258964945 +0000 UTC m=+2070.429930127" Oct 30 22:15:05.168018 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:05.167980 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:05.244631 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:05.244597 6720 generic.go:356] "Generic (PLEG): container finished" podID="d6b2208e-5fa5-4f8e-9fd3-c12ce229b361" containerID="7a929bcb4efc1d93eeca49ace8d07a33992b18d19b64ead7dd9635bf4a97f8b2" exitCode=60 Oct 30 22:15:05.244825 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:05.244688 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0f73c9-29889975-bpmgx" event={"ID":"d6b2208e-5fa5-4f8e-9fd3-c12ce229b361","Type":"ContainerDied","Data":"7a929bcb4efc1d93eeca49ace8d07a33992b18d19b64ead7dd9635bf4a97f8b2"} Oct 30 22:15:05.244825 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:05.244774 6720 scope.go:122] "RemoveContainer" containerID="74032b3e7bb51482c85a58c6a32fc938438e4985a7c5b89d8cd7838fa77fb4f9" Oct 30 22:15:05.244994 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:05.244977 6720 scope.go:122] "RemoveContainer" containerID="7a929bcb4efc1d93eeca49ace8d07a33992b18d19b64ead7dd9635bf4a97f8b2" Oct 30 22:15:05.245354 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:15:05.245332 6720 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-0f73c9-29889975-bpmgx_odh-ai-gateway-infra(d6b2208e-5fa5-4f8e-9fd3-c12ce229b361)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0f73c9-29889975-bpmgx" podUID="d6b2208e-5fa5-4f8e-9fd3-c12ce229b361" Oct 30 22:15:05.246921 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:05.246899 6720 generic.go:356] "Generic (PLEG): container finished" podID="ded6d4c9-cc20-4da2-bcb5-02f07193b223" containerID="37826089595cb5ca5b5bf101d4c58d5115a6fa35d9038e7f585bd89e2c77d704" exitCode=60 Oct 30 22:15:05.247024 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:05.246971 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6f8222-29889975-zb59n" event={"ID":"ded6d4c9-cc20-4da2-bcb5-02f07193b223","Type":"ContainerDied","Data":"37826089595cb5ca5b5bf101d4c58d5115a6fa35d9038e7f585bd89e2c77d704"} Oct 30 22:15:05.247225 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:05.247210 6720 scope.go:122] "RemoveContainer" containerID="37826089595cb5ca5b5bf101d4c58d5115a6fa35d9038e7f585bd89e2c77d704" Oct 30 22:15:05.247593 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:15:05.247567 6720 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-6f8222-29889975-zb59n_odh-ai-gateway-infra(ded6d4c9-cc20-4da2-bcb5-02f07193b223)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6f8222-29889975-zb59n" podUID="ded6d4c9-cc20-4da2-bcb5-02f07193b223" Oct 30 22:15:05.249103 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:05.249080 6720 generic.go:356] "Generic (PLEG): container finished" podID="f1c46564-6708-4ed7-912e-9adfce46553e" containerID="c79393a5740a881edbd0e232448e4479036f0cea02ed0178b76d495bbba7a317" exitCode=60 Oct 30 22:15:05.249216 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:05.249127 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bacea0-29889975-gpxpm" event={"ID":"f1c46564-6708-4ed7-912e-9adfce46553e","Type":"ContainerDied","Data":"c79393a5740a881edbd0e232448e4479036f0cea02ed0178b76d495bbba7a317"} Oct 30 22:15:05.249416 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:05.249368 6720 scope.go:122] "RemoveContainer" containerID="c79393a5740a881edbd0e232448e4479036f0cea02ed0178b76d495bbba7a317" Oct 30 22:15:05.249675 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:15:05.249658 6720 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-bacea0-29889975-gpxpm_odh-ai-gateway-infra(f1c46564-6708-4ed7-912e-9adfce46553e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bacea0-29889975-gpxpm" podUID="f1c46564-6708-4ed7-912e-9adfce46553e" Oct 30 22:15:05.257968 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:05.257948 6720 scope.go:122] "RemoveContainer" containerID="6c8795cca96f3c2a3f6fbf6c77fd15f7ec92dc970d125831369f5a4d0c33804c" Oct 30 22:15:05.271944 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:05.271925 6720 scope.go:122] "RemoveContainer" containerID="9961e00ac751bf83b5a556ed697a4faf37cc124ea69b5d1b0433ec16232b3899" Oct 30 22:15:05.892714 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:05.892681 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:06.259220 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:06.259142 6720 scope.go:122] "RemoveContainer" containerID="37826089595cb5ca5b5bf101d4c58d5115a6fa35d9038e7f585bd89e2c77d704" Oct 30 22:15:06.259695 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:15:06.259546 6720 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-6f8222-29889975-zb59n_odh-ai-gateway-infra(ded6d4c9-cc20-4da2-bcb5-02f07193b223)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6f8222-29889975-zb59n" podUID="ded6d4c9-cc20-4da2-bcb5-02f07193b223" Oct 30 22:15:06.261180 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:06.261152 6720 scope.go:122] "RemoveContainer" containerID="c79393a5740a881edbd0e232448e4479036f0cea02ed0178b76d495bbba7a317" Oct 30 22:15:06.261636 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:15:06.261587 6720 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-bacea0-29889975-gpxpm_odh-ai-gateway-infra(f1c46564-6708-4ed7-912e-9adfce46553e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bacea0-29889975-gpxpm" podUID="f1c46564-6708-4ed7-912e-9adfce46553e" Oct 30 22:15:06.264709 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:06.264686 6720 scope.go:122] "RemoveContainer" containerID="7a929bcb4efc1d93eeca49ace8d07a33992b18d19b64ead7dd9635bf4a97f8b2" Oct 30 22:15:06.265060 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:15:06.265037 6720 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-0f73c9-29889975-bpmgx_odh-ai-gateway-infra(d6b2208e-5fa5-4f8e-9fd3-c12ce229b361)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0f73c9-29889975-bpmgx" podUID="d6b2208e-5fa5-4f8e-9fd3-c12ce229b361" Oct 30 22:15:06.490869 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:06.488739 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:06.699278 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:06.699242 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:06.908034 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:06.908005 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:07.074321 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:07.074237 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:07.578967 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:07.578914 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:07.741412 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:07.735276 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:08.354596 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:08.354548 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:09.174140 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:09.174066 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:09.850995 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:09.848659 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:10.406069 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:15:10.406032 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 22:15:10.546308 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:10.546271 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:11.212788 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:11.212749 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:11.894538 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:11.894494 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:12.813420 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:12.813363 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:13.537454 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:13.536574 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:14.368160 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:14.368122 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:15.003925 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:15.003872 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:15.672333 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:15.672296 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:16.335066 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:16.335025 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:16.951225 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:16.951187 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:17.708365 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:17.708323 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:17.949637 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:17.949604 6720 scope.go:122] "RemoveContainer" containerID="37826089595cb5ca5b5bf101d4c58d5115a6fa35d9038e7f585bd89e2c77d704" Oct 30 22:15:17.949901 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:17.949882 6720 scope.go:122] "RemoveContainer" containerID="7a929bcb4efc1d93eeca49ace8d07a33992b18d19b64ead7dd9635bf4a97f8b2" Oct 30 22:15:18.372594 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:18.372524 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6f8222-29889975-zb59n" event={"ID":"ded6d4c9-cc20-4da2-bcb5-02f07193b223","Type":"ContainerStarted","Data":"fb15247f4e3822863a5dd3f0e31c80489fb324c244e6cdfa77fb13158116d92d"} Oct 30 22:15:18.584213 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:18.584117 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:18.949788 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:18.949750 6720 scope.go:122] "RemoveContainer" containerID="c79393a5740a881edbd0e232448e4479036f0cea02ed0178b76d495bbba7a317" Oct 30 22:15:19.328234 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:19.328157 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:19.382324 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:19.382288 6720 generic.go:356] "Generic (PLEG): container finished" podID="d6b2208e-5fa5-4f8e-9fd3-c12ce229b361" containerID="bc528595d085083d4380ec1e4e1c8ea94ee4d230f99e42cfceaa044aacfe90fa" exitCode=60 Oct 30 22:15:19.382324 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:19.382316 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0f73c9-29889975-bpmgx" event={"ID":"d6b2208e-5fa5-4f8e-9fd3-c12ce229b361","Type":"ContainerDied","Data":"bc528595d085083d4380ec1e4e1c8ea94ee4d230f99e42cfceaa044aacfe90fa"} Oct 30 22:15:19.382574 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:19.382429 6720 scope.go:122] "RemoveContainer" containerID="7a929bcb4efc1d93eeca49ace8d07a33992b18d19b64ead7dd9635bf4a97f8b2" Oct 30 22:15:19.382883 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:19.382851 6720 scope.go:122] "RemoveContainer" containerID="bc528595d085083d4380ec1e4e1c8ea94ee4d230f99e42cfceaa044aacfe90fa" Oct 30 22:15:19.383255 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:15:19.383230 6720 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-0f73c9-29889975-bpmgx_odh-ai-gateway-infra(d6b2208e-5fa5-4f8e-9fd3-c12ce229b361)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0f73c9-29889975-bpmgx" podUID="d6b2208e-5fa5-4f8e-9fd3-c12ce229b361" Oct 30 22:15:19.385058 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:19.385036 6720 generic.go:356] "Generic (PLEG): container finished" podID="ded6d4c9-cc20-4da2-bcb5-02f07193b223" containerID="fb15247f4e3822863a5dd3f0e31c80489fb324c244e6cdfa77fb13158116d92d" exitCode=60 Oct 30 22:15:19.385202 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:19.385080 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6f8222-29889975-zb59n" event={"ID":"ded6d4c9-cc20-4da2-bcb5-02f07193b223","Type":"ContainerDied","Data":"fb15247f4e3822863a5dd3f0e31c80489fb324c244e6cdfa77fb13158116d92d"} Oct 30 22:15:19.385453 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:19.385433 6720 scope.go:122] "RemoveContainer" containerID="fb15247f4e3822863a5dd3f0e31c80489fb324c244e6cdfa77fb13158116d92d" Oct 30 22:15:19.385849 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:15:19.385827 6720 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-6f8222-29889975-zb59n_odh-ai-gateway-infra(ded6d4c9-cc20-4da2-bcb5-02f07193b223)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6f8222-29889975-zb59n" podUID="ded6d4c9-cc20-4da2-bcb5-02f07193b223" Oct 30 22:15:19.400440 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:19.400366 6720 scope.go:122] "RemoveContainer" containerID="37826089595cb5ca5b5bf101d4c58d5115a6fa35d9038e7f585bd89e2c77d704" Oct 30 22:15:19.402918 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:19.402894 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6f8222-29889975-zb59n"] Oct 30 22:15:20.128078 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:20.121622 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:20.396191 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:20.396141 6720 generic.go:356] "Generic (PLEG): container finished" podID="f1c46564-6708-4ed7-912e-9adfce46553e" containerID="68b18e85f713a017ac4476f04dc56b98fbb375fd54260ca408fc3b721cf93f4a" exitCode=60 Oct 30 22:15:20.396331 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:20.396208 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bacea0-29889975-gpxpm" event={"ID":"f1c46564-6708-4ed7-912e-9adfce46553e","Type":"ContainerDied","Data":"68b18e85f713a017ac4476f04dc56b98fbb375fd54260ca408fc3b721cf93f4a"} Oct 30 22:15:20.396331 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:20.396281 6720 scope.go:122] "RemoveContainer" containerID="c79393a5740a881edbd0e232448e4479036f0cea02ed0178b76d495bbba7a317" Oct 30 22:15:20.396793 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:20.396774 6720 scope.go:122] "RemoveContainer" containerID="68b18e85f713a017ac4476f04dc56b98fbb375fd54260ca408fc3b721cf93f4a" Oct 30 22:15:20.397198 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:15:20.397176 6720 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-bacea0-29889975-gpxpm_odh-ai-gateway-infra(f1c46564-6708-4ed7-912e-9adfce46553e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bacea0-29889975-gpxpm" podUID="f1c46564-6708-4ed7-912e-9adfce46553e" Oct 30 22:15:20.416790 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:20.416759 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0f73c9-29889975-bpmgx"] Oct 30 22:15:20.604484 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:20.604448 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6f8222-29889975-zb59n" Oct 30 22:15:20.656747 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:20.656674 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0f73c9-29889975-bpmgx" Oct 30 22:15:20.730901 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:20.730864 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ded6d4c9-cc20-4da2-bcb5-02f07193b223-kube-api-access-mxzbq\" (UniqueName: \"kubernetes.io/projected/ded6d4c9-cc20-4da2-bcb5-02f07193b223-kube-api-access-mxzbq\") pod \"ded6d4c9-cc20-4da2-bcb5-02f07193b223\" (UID: \"ded6d4c9-cc20-4da2-bcb5-02f07193b223\") " Oct 30 22:15:20.731111 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:20.730935 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ded6d4c9-cc20-4da2-bcb5-02f07193b223-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ded6d4c9-cc20-4da2-bcb5-02f07193b223-maas-api-service-ca\") pod \"ded6d4c9-cc20-4da2-bcb5-02f07193b223\" (UID: \"ded6d4c9-cc20-4da2-bcb5-02f07193b223\") " Oct 30 22:15:20.731206 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:20.731182 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6b2208e-5fa5-4f8e-9fd3-c12ce229b361-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6b2208e-5fa5-4f8e-9fd3-c12ce229b361-maas-api-service-ca\") pod \"d6b2208e-5fa5-4f8e-9fd3-c12ce229b361\" (UID: \"d6b2208e-5fa5-4f8e-9fd3-c12ce229b361\") " Oct 30 22:15:20.731339 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:20.731322 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6b2208e-5fa5-4f8e-9fd3-c12ce229b361-kube-api-access-9mwgf\" (UniqueName: \"kubernetes.io/projected/d6b2208e-5fa5-4f8e-9fd3-c12ce229b361-kube-api-access-9mwgf\") pod \"d6b2208e-5fa5-4f8e-9fd3-c12ce229b361\" (UID: \"d6b2208e-5fa5-4f8e-9fd3-c12ce229b361\") " Oct 30 22:15:20.731560 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:20.731527 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ded6d4c9-cc20-4da2-bcb5-02f07193b223-maas-api-service-ca" pod "ded6d4c9-cc20-4da2-bcb5-02f07193b223" (UID: "ded6d4c9-cc20-4da2-bcb5-02f07193b223"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:20.731895 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:20.731871 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6b2208e-5fa5-4f8e-9fd3-c12ce229b361-maas-api-service-ca" pod "d6b2208e-5fa5-4f8e-9fd3-c12ce229b361" (UID: "d6b2208e-5fa5-4f8e-9fd3-c12ce229b361"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:20.732059 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:20.732042 6720 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6b2208e-5fa5-4f8e-9fd3-c12ce229b361-maas-api-service-ca\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:20.732118 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:20.732066 6720 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ded6d4c9-cc20-4da2-bcb5-02f07193b223-maas-api-service-ca\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:20.734886 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:20.734860 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ded6d4c9-cc20-4da2-bcb5-02f07193b223-kube-api-access-mxzbq" pod "ded6d4c9-cc20-4da2-bcb5-02f07193b223" (UID: "ded6d4c9-cc20-4da2-bcb5-02f07193b223"). InnerVolumeSpecName "kube-api-access-mxzbq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:20.741803 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:20.741779 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6b2208e-5fa5-4f8e-9fd3-c12ce229b361-kube-api-access-9mwgf" pod "d6b2208e-5fa5-4f8e-9fd3-c12ce229b361" (UID: "d6b2208e-5fa5-4f8e-9fd3-c12ce229b361"). InnerVolumeSpecName "kube-api-access-9mwgf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:20.836269 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:20.836224 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:20.838355 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:20.838323 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxzbq\" (UniqueName: \"kubernetes.io/projected/ded6d4c9-cc20-4da2-bcb5-02f07193b223-kube-api-access-mxzbq\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:20.838355 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:20.838359 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mwgf\" (UniqueName: \"kubernetes.io/projected/d6b2208e-5fa5-4f8e-9fd3-c12ce229b361-kube-api-access-9mwgf\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:21.407129 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:21.407101 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0f73c9-29889975-bpmgx" Oct 30 22:15:21.407616 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:21.407141 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0f73c9-29889975-bpmgx" event={"ID":"d6b2208e-5fa5-4f8e-9fd3-c12ce229b361","Type":"ContainerDied","Data":"7eac2058be8dba15ef5aa2a804fb1221feb36ae01e239669f0d196b33718231f"} Oct 30 22:15:21.407616 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:21.407211 6720 scope.go:122] "RemoveContainer" containerID="bc528595d085083d4380ec1e4e1c8ea94ee4d230f99e42cfceaa044aacfe90fa" Oct 30 22:15:21.409234 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:21.409211 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6f8222-29889975-zb59n" Oct 30 22:15:21.409347 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:21.409209 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6f8222-29889975-zb59n" event={"ID":"ded6d4c9-cc20-4da2-bcb5-02f07193b223","Type":"ContainerDied","Data":"97871b31bf5ce62d15bac8a397e9646c74f5ef6bd101484e83db966e2d4e3a69"} Oct 30 22:15:21.419833 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:21.419812 6720 scope.go:122] "RemoveContainer" containerID="fb15247f4e3822863a5dd3f0e31c80489fb324c244e6cdfa77fb13158116d92d" Oct 30 22:15:21.443478 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:21.443452 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0f73c9-29889975-bpmgx"] Oct 30 22:15:21.447056 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:21.447034 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0f73c9-29889975-bpmgx"] Oct 30 22:15:21.461563 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:21.461544 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6f8222-29889975-zb59n"] Oct 30 22:15:21.465290 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:21.465269 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-6f8222-29889975-zb59n"] Oct 30 22:15:21.484607 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:21.484580 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bacea0-29889975-gpxpm"] Oct 30 22:15:21.552172 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:21.551744 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:21.698185 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:21.698161 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bacea0-29889975-gpxpm" Oct 30 22:15:21.856702 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:21.856671 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1c46564-6708-4ed7-912e-9adfce46553e-kube-api-access-sx676\" (UniqueName: \"kubernetes.io/projected/f1c46564-6708-4ed7-912e-9adfce46553e-kube-api-access-sx676\") pod \"f1c46564-6708-4ed7-912e-9adfce46553e\" (UID: \"f1c46564-6708-4ed7-912e-9adfce46553e\") " Oct 30 22:15:21.857132 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:21.856746 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1c46564-6708-4ed7-912e-9adfce46553e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1c46564-6708-4ed7-912e-9adfce46553e-maas-api-service-ca\") pod \"f1c46564-6708-4ed7-912e-9adfce46553e\" (UID: \"f1c46564-6708-4ed7-912e-9adfce46553e\") " Oct 30 22:15:21.857241 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:21.857126 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1c46564-6708-4ed7-912e-9adfce46553e-maas-api-service-ca" pod "f1c46564-6708-4ed7-912e-9adfce46553e" (UID: "f1c46564-6708-4ed7-912e-9adfce46553e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:21.857450 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:21.857433 6720 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1c46564-6708-4ed7-912e-9adfce46553e-maas-api-service-ca\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:21.859114 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:21.859091 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1c46564-6708-4ed7-912e-9adfce46553e-kube-api-access-sx676" pod "f1c46564-6708-4ed7-912e-9adfce46553e" (UID: "f1c46564-6708-4ed7-912e-9adfce46553e"). InnerVolumeSpecName "kube-api-access-sx676". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:21.957646 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:21.957560 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6b2208e-5fa5-4f8e-9fd3-c12ce229b361" path="/var/lib/kubelet/pods/d6b2208e-5fa5-4f8e-9fd3-c12ce229b361/volumes" Oct 30 22:15:21.958074 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:21.958059 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ded6d4c9-cc20-4da2-bcb5-02f07193b223" path="/var/lib/kubelet/pods/ded6d4c9-cc20-4da2-bcb5-02f07193b223/volumes" Oct 30 22:15:21.958532 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:21.958501 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sx676\" (UniqueName: \"kubernetes.io/projected/f1c46564-6708-4ed7-912e-9adfce46553e-kube-api-access-sx676\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:22.266269 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:22.266167 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:22.421536 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:22.421501 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bacea0-29889975-gpxpm" event={"ID":"f1c46564-6708-4ed7-912e-9adfce46553e","Type":"ContainerDied","Data":"86ad8bc172ac7bb5a2fc4308b45a99ebd4dabb1440374b72579d20950695a19d"} Oct 30 22:15:22.422558 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:22.421563 6720 scope.go:122] "RemoveContainer" containerID="68b18e85f713a017ac4476f04dc56b98fbb375fd54260ca408fc3b721cf93f4a" Oct 30 22:15:22.422558 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:22.421684 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bacea0-29889975-gpxpm" Oct 30 22:15:22.456346 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:22.456314 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bacea0-29889975-gpxpm"] Oct 30 22:15:22.459199 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:22.459172 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bacea0-29889975-gpxpm"] Oct 30 22:15:23.186445 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:23.186410 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:23.939839 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:23.939805 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:23.988210 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:23.988169 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1c46564-6708-4ed7-912e-9adfce46553e" path="/var/lib/kubelet/pods/f1c46564-6708-4ed7-912e-9adfce46553e/volumes" Oct 30 22:15:24.910328 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:24.910300 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:25.942073 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:25.942029 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:26.628243 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:26.628203 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:27.255627 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:27.254147 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:27.970576 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:27.962928 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:28.697364 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:28.697331 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:28.971586 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:28.971488 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:29.326466 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:29.324217 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:30.120107 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:30.120072 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:30.759524 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:30.759464 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:30.925419 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:30.923646 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:31.158312 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:31.158281 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:31.345796 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:31.344432 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:31.539187 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:31.539103 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:32.105797 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:32.105757 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:32.761561 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:32.761520 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:33.514294 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:33.513888 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:34.318406 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:34.317553 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:35.025561 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:35.025464 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:35.651518 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:35.650914 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:36.382337 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:36.382301 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:37.037104 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:37.037049 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:37.750353 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:37.750200 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:38.104769 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:38.104709 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:38.269931 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:38.269867 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:38.443967 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:38.443829 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:38.619486 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:38.619435 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:39.213653 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:39.211894 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:40.030946 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:40.030905 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:40.407539 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:15:40.407504 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 22:15:40.715612 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:40.715492 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:41.356294 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:41.356154 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:42.051359 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:42.051319 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:42.834830 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:42.834792 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:43.443415 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:43.443358 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:44.239855 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:44.239768 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:44.538096 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:44.537999 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:44.677036 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:44.677001 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:44.926431 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:44.926370 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:45.101934 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:45.101894 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:45.252322 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:45.252242 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:45.869489 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:45.869429 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:46.193050 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:46.192937 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:46.853684 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:46.853609 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:47.712589 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:47.712534 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:48.367350 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:48.367316 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:49.027990 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:49.027949 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:49.725226 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:49.725188 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:50.462448 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:50.462379 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:51.100910 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:51.100862 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:51.813204 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:51.813168 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:52.483713 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:52.483661 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:53.248778 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:53.248741 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:54.104668 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:54.104631 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:54.998885 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:54.998854 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:55.621164 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:55.621125 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:55.789148 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:55.789092 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:56.311215 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:56.311176 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:56.540360 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:56.540314 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:56.704499 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:56.704456 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:56.856869 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:56.856834 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:57.615429 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:57.615365 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:58.428809 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:58.428750 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:59.230641 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:59.230605 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:15:59.923990 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:15:59.923951 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:00.552069 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:00.551013 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:01.380874 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:01.380836 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:01.943176 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:01.943077 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:02.090346 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:02.090236 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:02.500068 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:02.500034 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:03.495587 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:03.495557 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:04.193128 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:04.191363 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:04.562945 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:04.562855 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:04.709890 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:04.709860 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:04.938877 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:04.937140 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-6brww"] Oct 30 22:16:04.938877 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:04.938311 6720 cpu_manager.go:414] "containerMap: removing container" podUID="ded6d4c9-cc20-4da2-bcb5-02f07193b223" containerName="cleanup" Oct 30 22:16:04.938877 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:04.938333 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ded6d4c9-cc20-4da2-bcb5-02f07193b223" containerName="cleanup" Oct 30 22:16:04.938877 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:04.938359 6720 cpu_manager.go:414] "containerMap: removing container" podUID="f1c46564-6708-4ed7-912e-9adfce46553e" containerName="cleanup" Oct 30 22:16:04.938877 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:04.938366 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1c46564-6708-4ed7-912e-9adfce46553e" containerName="cleanup" Oct 30 22:16:04.938877 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:04.938420 6720 cpu_manager.go:414] "containerMap: removing container" podUID="d6b2208e-5fa5-4f8e-9fd3-c12ce229b361" containerName="cleanup" Oct 30 22:16:04.938877 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:04.938428 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6b2208e-5fa5-4f8e-9fd3-c12ce229b361" containerName="cleanup" Oct 30 22:16:04.938877 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:04.938448 6720 cpu_manager.go:414] "containerMap: removing container" podUID="d6b2208e-5fa5-4f8e-9fd3-c12ce229b361" containerName="cleanup" Oct 30 22:16:04.938877 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:04.938456 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6b2208e-5fa5-4f8e-9fd3-c12ce229b361" containerName="cleanup" Oct 30 22:16:04.938877 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:04.938472 6720 cpu_manager.go:414] "containerMap: removing container" podUID="ded6d4c9-cc20-4da2-bcb5-02f07193b223" containerName="cleanup" Oct 30 22:16:04.938877 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:04.938479 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ded6d4c9-cc20-4da2-bcb5-02f07193b223" containerName="cleanup" Oct 30 22:16:04.938877 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:04.938494 6720 cpu_manager.go:414] "containerMap: removing container" podUID="f1c46564-6708-4ed7-912e-9adfce46553e" containerName="cleanup" Oct 30 22:16:04.938877 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:04.938501 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1c46564-6708-4ed7-912e-9adfce46553e" containerName="cleanup" Oct 30 22:16:04.938877 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:04.938675 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6b2208e-5fa5-4f8e-9fd3-c12ce229b361" containerName="cleanup" Oct 30 22:16:04.938877 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:04.938691 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6b2208e-5fa5-4f8e-9fd3-c12ce229b361" containerName="cleanup" Oct 30 22:16:04.938877 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:04.938702 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="ded6d4c9-cc20-4da2-bcb5-02f07193b223" containerName="cleanup" Oct 30 22:16:04.938877 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:04.938711 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1c46564-6708-4ed7-912e-9adfce46553e" containerName="cleanup" Oct 30 22:16:04.938877 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:04.938735 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1c46564-6708-4ed7-912e-9adfce46553e" containerName="cleanup" Oct 30 22:16:04.938877 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:04.938746 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="ded6d4c9-cc20-4da2-bcb5-02f07193b223" containerName="cleanup" Oct 30 22:16:04.940065 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:04.939835 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:04.943690 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:04.943658 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-6brww" Oct 30 22:16:04.949822 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:04.949777 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-6brww"] Oct 30 22:16:04.957559 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:04.957514 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-a39158-openshift-default-dockercfg-xz2kf\"" Oct 30 22:16:05.068154 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:05.068105 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8566ec43-9f32-4ad1-a46f-fc94ccf1b24d-workload-socket\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-6brww\" (UID: \"8566ec43-9f32-4ad1-a46f-fc94ccf1b24d\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-6brww" Oct 30 22:16:05.068361 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:05.068289 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8566ec43-9f32-4ad1-a46f-fc94ccf1b24d-istio-envoy\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-6brww\" (UID: \"8566ec43-9f32-4ad1-a46f-fc94ccf1b24d\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-6brww" Oct 30 22:16:05.068361 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:05.068332 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6czh\" (UniqueName: \"kubernetes.io/projected/8566ec43-9f32-4ad1-a46f-fc94ccf1b24d-kube-api-access-w6czh\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-6brww\" (UID: \"8566ec43-9f32-4ad1-a46f-fc94ccf1b24d\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-6brww" Oct 30 22:16:05.068595 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:05.068364 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8566ec43-9f32-4ad1-a46f-fc94ccf1b24d-credential-socket\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-6brww\" (UID: \"8566ec43-9f32-4ad1-a46f-fc94ccf1b24d\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-6brww" Oct 30 22:16:05.068595 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:05.068456 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8566ec43-9f32-4ad1-a46f-fc94ccf1b24d-istiod-ca-cert\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-6brww\" (UID: \"8566ec43-9f32-4ad1-a46f-fc94ccf1b24d\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-6brww" Oct 30 22:16:05.068595 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:05.068549 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8566ec43-9f32-4ad1-a46f-fc94ccf1b24d-istio-token\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-6brww\" (UID: \"8566ec43-9f32-4ad1-a46f-fc94ccf1b24d\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-6brww" Oct 30 22:16:05.068756 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:05.068648 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8566ec43-9f32-4ad1-a46f-fc94ccf1b24d-workload-certs\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-6brww\" (UID: \"8566ec43-9f32-4ad1-a46f-fc94ccf1b24d\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-6brww" Oct 30 22:16:05.068756 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:05.068714 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8566ec43-9f32-4ad1-a46f-fc94ccf1b24d-istio-data\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-6brww\" (UID: \"8566ec43-9f32-4ad1-a46f-fc94ccf1b24d\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-6brww" Oct 30 22:16:05.068856 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:05.068752 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8566ec43-9f32-4ad1-a46f-fc94ccf1b24d-istio-podinfo\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-6brww\" (UID: \"8566ec43-9f32-4ad1-a46f-fc94ccf1b24d\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-6brww" Oct 30 22:16:05.170486 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:05.170451 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8566ec43-9f32-4ad1-a46f-fc94ccf1b24d-workload-socket\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-6brww\" (UID: \"8566ec43-9f32-4ad1-a46f-fc94ccf1b24d\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-6brww" Oct 30 22:16:05.170697 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:05.170545 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8566ec43-9f32-4ad1-a46f-fc94ccf1b24d-istio-envoy\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-6brww\" (UID: \"8566ec43-9f32-4ad1-a46f-fc94ccf1b24d\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-6brww" Oct 30 22:16:05.170697 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:05.170567 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6czh\" (UniqueName: \"kubernetes.io/projected/8566ec43-9f32-4ad1-a46f-fc94ccf1b24d-kube-api-access-w6czh\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-6brww\" (UID: \"8566ec43-9f32-4ad1-a46f-fc94ccf1b24d\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-6brww" Oct 30 22:16:05.170697 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:05.170588 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8566ec43-9f32-4ad1-a46f-fc94ccf1b24d-credential-socket\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-6brww\" (UID: \"8566ec43-9f32-4ad1-a46f-fc94ccf1b24d\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-6brww" Oct 30 22:16:05.170697 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:05.170620 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8566ec43-9f32-4ad1-a46f-fc94ccf1b24d-istiod-ca-cert\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-6brww\" (UID: \"8566ec43-9f32-4ad1-a46f-fc94ccf1b24d\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-6brww" Oct 30 22:16:05.170926 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:05.170698 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8566ec43-9f32-4ad1-a46f-fc94ccf1b24d-istio-token\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-6brww\" (UID: \"8566ec43-9f32-4ad1-a46f-fc94ccf1b24d\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-6brww" Oct 30 22:16:05.170926 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:05.170793 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8566ec43-9f32-4ad1-a46f-fc94ccf1b24d-workload-certs\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-6brww\" (UID: \"8566ec43-9f32-4ad1-a46f-fc94ccf1b24d\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-6brww" Oct 30 22:16:05.170926 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:05.170852 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8566ec43-9f32-4ad1-a46f-fc94ccf1b24d-istio-data\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-6brww\" (UID: \"8566ec43-9f32-4ad1-a46f-fc94ccf1b24d\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-6brww" Oct 30 22:16:05.170926 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:05.170881 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8566ec43-9f32-4ad1-a46f-fc94ccf1b24d-istio-podinfo\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-6brww\" (UID: \"8566ec43-9f32-4ad1-a46f-fc94ccf1b24d\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-6brww" Oct 30 22:16:05.171138 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:05.170931 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8566ec43-9f32-4ad1-a46f-fc94ccf1b24d-workload-socket\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-6brww\" (UID: \"8566ec43-9f32-4ad1-a46f-fc94ccf1b24d\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-6brww" Oct 30 22:16:05.171303 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:05.171281 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8566ec43-9f32-4ad1-a46f-fc94ccf1b24d-credential-socket\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-6brww\" (UID: \"8566ec43-9f32-4ad1-a46f-fc94ccf1b24d\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-6brww" Oct 30 22:16:05.171607 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:05.171578 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8566ec43-9f32-4ad1-a46f-fc94ccf1b24d-istio-data\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-6brww\" (UID: \"8566ec43-9f32-4ad1-a46f-fc94ccf1b24d\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-6brww" Oct 30 22:16:05.171740 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:05.171633 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8566ec43-9f32-4ad1-a46f-fc94ccf1b24d-workload-certs\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-6brww\" (UID: \"8566ec43-9f32-4ad1-a46f-fc94ccf1b24d\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-6brww" Oct 30 22:16:05.171871 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:05.171845 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8566ec43-9f32-4ad1-a46f-fc94ccf1b24d-istiod-ca-cert\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-6brww\" (UID: \"8566ec43-9f32-4ad1-a46f-fc94ccf1b24d\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-6brww" Oct 30 22:16:05.173873 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:05.173828 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8566ec43-9f32-4ad1-a46f-fc94ccf1b24d-istio-envoy\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-6brww\" (UID: \"8566ec43-9f32-4ad1-a46f-fc94ccf1b24d\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-6brww" Oct 30 22:16:05.176557 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:05.176531 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8566ec43-9f32-4ad1-a46f-fc94ccf1b24d-istio-podinfo\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-6brww\" (UID: \"8566ec43-9f32-4ad1-a46f-fc94ccf1b24d\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-6brww" Oct 30 22:16:05.185731 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:05.185705 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8566ec43-9f32-4ad1-a46f-fc94ccf1b24d-istio-token\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-6brww\" (UID: \"8566ec43-9f32-4ad1-a46f-fc94ccf1b24d\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-6brww" Oct 30 22:16:05.185842 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:05.185707 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6czh\" (UniqueName: \"kubernetes.io/projected/8566ec43-9f32-4ad1-a46f-fc94ccf1b24d-kube-api-access-w6czh\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-6brww\" (UID: \"8566ec43-9f32-4ad1-a46f-fc94ccf1b24d\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-6brww" Oct 30 22:16:05.271108 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:05.271019 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-6brww" Oct 30 22:16:05.566624 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:05.566588 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-6brww"] Oct 30 22:16:05.569613 ip-10-0-137-181 kubenswrapper[6720]: W1030 22:16:05.569584 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8566ec43_9f32_4ad1_a46f_fc94ccf1b24d.slice/crio-55e43d638b333b18715136592b076adc9f8b6189672ae3c7d2a4ac859d5a5e59 WatchSource:0}: Error finding container 55e43d638b333b18715136592b076adc9f8b6189672ae3c7d2a4ac859d5a5e59: Status 404 returned error can't find the container with id 55e43d638b333b18715136592b076adc9f8b6189672ae3c7d2a4ac859d5a5e59 Oct 30 22:16:05.574068 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:05.574027 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:16:05.574167 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:05.574122 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:16:05.574208 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:05.574165 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:16:05.734898 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:05.734850 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:05.800921 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:05.800884 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-6brww" event={"ID":"8566ec43-9f32-4ad1-a46f-fc94ccf1b24d","Type":"ContainerStarted","Data":"c12034b9c83b2e5b8972284dde2537e89afc73effffc3b2df7b4672b876a74ad"} Oct 30 22:16:05.801222 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:05.801189 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-6brww" event={"ID":"8566ec43-9f32-4ad1-a46f-fc94ccf1b24d","Type":"ContainerStarted","Data":"55e43d638b333b18715136592b076adc9f8b6189672ae3c7d2a4ac859d5a5e59"} Oct 30 22:16:05.826505 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:05.826367 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-6brww" podStartSLOduration=1.826348663 podStartE2EDuration="1.826348663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:05.82524603 +0000 UTC m=+2131.996211211" watchObservedRunningTime="2026-10-30 22:16:05.826348663 +0000 UTC m=+2131.997313848" Oct 30 22:16:06.095479 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:06.095299 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:06.266418 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:06.240868 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:06.288449 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:06.274800 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-6brww" Oct 30 22:16:06.288449 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:06.275997 6720 patch_prober.go:29] interesting pod/e2e-extmod-a-a39158-openshift-default-575c648d9-6brww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 30 22:16:06.288449 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:06.276049 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-6brww" podUID="8566ec43-9f32-4ad1-a46f-fc94ccf1b24d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 30 22:16:06.398049 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:06.398012 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:06.787620 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:06.786220 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:06.967908 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:06.967829 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:07.145616 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:07.145448 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:07.272767 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:07.272729 6720 patch_prober.go:29] interesting pod/e2e-extmod-a-a39158-openshift-default-575c648d9-6brww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 30 22:16:07.273082 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:07.273041 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-6brww" podUID="8566ec43-9f32-4ad1-a46f-fc94ccf1b24d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 30 22:16:07.322700 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:07.322670 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:07.581713 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:07.577408 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:08.233149 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:08.233110 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:08.285952 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:08.285919 6720 patch_prober.go:29] interesting pod/e2e-extmod-a-a39158-openshift-default-575c648d9-6brww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 30 22:16:08.286294 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:08.286243 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-6brww" podUID="8566ec43-9f32-4ad1-a46f-fc94ccf1b24d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 30 22:16:08.421413 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:08.420573 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:08.871413 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:08.864268 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:09.016998 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:09.016969 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:09.277060 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:09.276972 6720 patch_prober.go:29] interesting pod/e2e-extmod-a-a39158-openshift-default-575c648d9-6brww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 30 22:16:09.307564 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:09.307523 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-6brww" podUID="8566ec43-9f32-4ad1-a46f-fc94ccf1b24d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 30 22:16:09.986840 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:09.986802 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:10.277467 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:10.276910 6720 patch_prober.go:29] interesting pod/e2e-extmod-a-a39158-openshift-default-575c648d9-6brww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 30 22:16:10.277467 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:10.276990 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-6brww" podUID="8566ec43-9f32-4ad1-a46f-fc94ccf1b24d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 30 22:16:10.411406 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:16:10.408639 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 22:16:10.928407 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:10.927932 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:11.272530 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:11.271374 6720 patch_prober.go:29] interesting pod/e2e-extmod-a-a39158-openshift-default-575c648d9-6brww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 30 22:16:11.272530 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:11.271457 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-6brww" podUID="8566ec43-9f32-4ad1-a46f-fc94ccf1b24d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 30 22:16:11.934283 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:11.934246 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:12.272336 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:12.272257 6720 patch_prober.go:29] interesting pod/e2e-extmod-a-a39158-openshift-default-575c648d9-6brww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 30 22:16:12.272336 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:12.272316 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-6brww" podUID="8566ec43-9f32-4ad1-a46f-fc94ccf1b24d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 30 22:16:12.737108 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:12.737054 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:13.277044 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:13.277016 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-6brww" Oct 30 22:16:13.277531 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:13.277511 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-6brww" Oct 30 22:16:13.278219 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:13.278198 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-6brww" Oct 30 22:16:13.356706 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:13.355937 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:14.162864 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:14.161334 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:14.917171 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:14.917095 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:15.753280 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:15.753250 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:16.425628 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:16.425593 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:17.059358 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:17.057834 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:17.725171 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:17.724899 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:18.502335 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:18.502284 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:19.076140 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:19.076106 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:19.722508 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:19.722472 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:20.427435 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:20.424940 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:21.212183 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:21.212134 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:21.902645 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:21.899022 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:22.512179 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:22.512142 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:23.213921 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:23.213350 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:23.946400 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:23.946344 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:24.544099 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:24.544036 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:25.229938 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:25.227545 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:25.936211 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:25.936169 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:26.616837 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:26.616792 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:27.355665 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:27.355576 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:27.926644 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:27.926605 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:28.080335 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:28.080305 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:28.604104 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:28.604051 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:29.023541 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:29.023462 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:29.242852 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:29.242814 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:29.699004 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:29.698950 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:30.528374 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:30.528335 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:31.217467 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:31.217427 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:32.066434 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:32.066375 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:32.812619 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:32.812582 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:33.269018 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:33.268984 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:33.486589 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:33.486552 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:33.795008 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:33.794970 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:34.804083 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:34.804028 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:35.037257 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:35.037223 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:35.589585 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:35.589553 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:36.217271 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:36.217216 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:36.948406 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:36.947495 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:37.807805 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:37.807769 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:38.523302 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:38.523264 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:39.125996 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:39.119865 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:39.836030 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:39.835995 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:40.404528 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:16:40.404485 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 22:16:40.546551 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:40.546463 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:41.151739 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:41.151701 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:41.309211 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:41.309169 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:42.023865 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:42.023834 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:42.686141 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:42.686099 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:43.328722 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:43.328689 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:44.034673 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:44.034637 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:44.693414 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:44.691225 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:45.318899 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:45.318863 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:46.073119 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:46.072974 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:46.742763 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:46.742711 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:47.443620 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:47.443515 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:48.205451 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:48.205366 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:49.116806 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:49.116729 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:49.853783 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:49.853729 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:49.984269 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:49.984233 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-n5hzj"] Oct 30 22:16:49.985165 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:49.985142 6720 cpu_manager.go:414] "containerMap: removing container" podUID="d6b2208e-5fa5-4f8e-9fd3-c12ce229b361" containerName="cleanup" Oct 30 22:16:49.985234 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:49.985171 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6b2208e-5fa5-4f8e-9fd3-c12ce229b361" containerName="cleanup" Oct 30 22:16:49.985234 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:49.985192 6720 cpu_manager.go:414] "containerMap: removing container" podUID="ded6d4c9-cc20-4da2-bcb5-02f07193b223" containerName="cleanup" Oct 30 22:16:49.985234 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:49.985201 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ded6d4c9-cc20-4da2-bcb5-02f07193b223" containerName="cleanup" Oct 30 22:16:49.985234 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:49.985218 6720 cpu_manager.go:414] "containerMap: removing container" podUID="f1c46564-6708-4ed7-912e-9adfce46553e" containerName="cleanup" Oct 30 22:16:49.985234 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:49.985225 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1c46564-6708-4ed7-912e-9adfce46553e" containerName="cleanup" Oct 30 22:16:49.985434 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:49.985411 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6b2208e-5fa5-4f8e-9fd3-c12ce229b361" containerName="cleanup" Oct 30 22:16:49.985434 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:49.985427 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="ded6d4c9-cc20-4da2-bcb5-02f07193b223" containerName="cleanup" Oct 30 22:16:49.985497 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:49.985436 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1c46564-6708-4ed7-912e-9adfce46553e" containerName="cleanup" Oct 30 22:16:49.990070 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:49.990024 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-n5hzj" Oct 30 22:16:50.014824 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:50.014790 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-n5hzj"] Oct 30 22:16:50.028941 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:50.028901 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28b34a7e-4765-46d0-b412-74eb9a1567d9-credential-socket\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-n5hzj\" (UID: \"28b34a7e-4765-46d0-b412-74eb9a1567d9\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-n5hzj" Oct 30 22:16:50.029117 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:50.028963 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28b34a7e-4765-46d0-b412-74eb9a1567d9-workload-certs\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-n5hzj\" (UID: \"28b34a7e-4765-46d0-b412-74eb9a1567d9\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-n5hzj" Oct 30 22:16:50.029117 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:50.029029 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28b34a7e-4765-46d0-b412-74eb9a1567d9-istio-podinfo\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-n5hzj\" (UID: \"28b34a7e-4765-46d0-b412-74eb9a1567d9\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-n5hzj" Oct 30 22:16:50.029117 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:50.029083 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28b34a7e-4765-46d0-b412-74eb9a1567d9-istio-envoy\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-n5hzj\" (UID: \"28b34a7e-4765-46d0-b412-74eb9a1567d9\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-n5hzj" Oct 30 22:16:50.029309 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:50.029130 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28b34a7e-4765-46d0-b412-74eb9a1567d9-istiod-ca-cert\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-n5hzj\" (UID: \"28b34a7e-4765-46d0-b412-74eb9a1567d9\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-n5hzj" Oct 30 22:16:50.029309 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:50.029164 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4k7v\" (UniqueName: \"kubernetes.io/projected/28b34a7e-4765-46d0-b412-74eb9a1567d9-kube-api-access-k4k7v\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-n5hzj\" (UID: \"28b34a7e-4765-46d0-b412-74eb9a1567d9\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-n5hzj" Oct 30 22:16:50.029309 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:50.029187 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28b34a7e-4765-46d0-b412-74eb9a1567d9-workload-socket\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-n5hzj\" (UID: \"28b34a7e-4765-46d0-b412-74eb9a1567d9\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-n5hzj" Oct 30 22:16:50.029309 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:50.029218 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28b34a7e-4765-46d0-b412-74eb9a1567d9-istio-data\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-n5hzj\" (UID: \"28b34a7e-4765-46d0-b412-74eb9a1567d9\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-n5hzj" Oct 30 22:16:50.029309 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:50.029249 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28b34a7e-4765-46d0-b412-74eb9a1567d9-istio-token\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-n5hzj\" (UID: \"28b34a7e-4765-46d0-b412-74eb9a1567d9\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-n5hzj" Oct 30 22:16:50.130336 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:50.130251 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28b34a7e-4765-46d0-b412-74eb9a1567d9-istio-podinfo\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-n5hzj\" (UID: \"28b34a7e-4765-46d0-b412-74eb9a1567d9\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-n5hzj" Oct 30 22:16:50.130819 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:50.130339 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28b34a7e-4765-46d0-b412-74eb9a1567d9-istio-envoy\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-n5hzj\" (UID: \"28b34a7e-4765-46d0-b412-74eb9a1567d9\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-n5hzj" Oct 30 22:16:50.130819 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:50.130444 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28b34a7e-4765-46d0-b412-74eb9a1567d9-istiod-ca-cert\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-n5hzj\" (UID: \"28b34a7e-4765-46d0-b412-74eb9a1567d9\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-n5hzj" Oct 30 22:16:50.130819 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:50.130495 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4k7v\" (UniqueName: \"kubernetes.io/projected/28b34a7e-4765-46d0-b412-74eb9a1567d9-kube-api-access-k4k7v\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-n5hzj\" (UID: \"28b34a7e-4765-46d0-b412-74eb9a1567d9\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-n5hzj" Oct 30 22:16:50.130819 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:50.130597 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28b34a7e-4765-46d0-b412-74eb9a1567d9-workload-socket\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-n5hzj\" (UID: \"28b34a7e-4765-46d0-b412-74eb9a1567d9\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-n5hzj" Oct 30 22:16:50.130819 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:50.130677 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28b34a7e-4765-46d0-b412-74eb9a1567d9-istio-data\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-n5hzj\" (UID: \"28b34a7e-4765-46d0-b412-74eb9a1567d9\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-n5hzj" Oct 30 22:16:50.130819 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:50.130737 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28b34a7e-4765-46d0-b412-74eb9a1567d9-istio-token\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-n5hzj\" (UID: \"28b34a7e-4765-46d0-b412-74eb9a1567d9\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-n5hzj" Oct 30 22:16:50.131142 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:50.130822 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28b34a7e-4765-46d0-b412-74eb9a1567d9-credential-socket\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-n5hzj\" (UID: \"28b34a7e-4765-46d0-b412-74eb9a1567d9\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-n5hzj" Oct 30 22:16:50.131142 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:50.130876 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28b34a7e-4765-46d0-b412-74eb9a1567d9-workload-certs\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-n5hzj\" (UID: \"28b34a7e-4765-46d0-b412-74eb9a1567d9\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-n5hzj" Oct 30 22:16:50.131142 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:50.130964 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28b34a7e-4765-46d0-b412-74eb9a1567d9-workload-socket\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-n5hzj\" (UID: \"28b34a7e-4765-46d0-b412-74eb9a1567d9\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-n5hzj" Oct 30 22:16:50.131142 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:50.131039 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28b34a7e-4765-46d0-b412-74eb9a1567d9-istio-data\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-n5hzj\" (UID: \"28b34a7e-4765-46d0-b412-74eb9a1567d9\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-n5hzj" Oct 30 22:16:50.131365 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:50.131219 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28b34a7e-4765-46d0-b412-74eb9a1567d9-workload-certs\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-n5hzj\" (UID: \"28b34a7e-4765-46d0-b412-74eb9a1567d9\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-n5hzj" Oct 30 22:16:50.131365 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:50.131311 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28b34a7e-4765-46d0-b412-74eb9a1567d9-credential-socket\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-n5hzj\" (UID: \"28b34a7e-4765-46d0-b412-74eb9a1567d9\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-n5hzj" Oct 30 22:16:50.131472 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:50.131417 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28b34a7e-4765-46d0-b412-74eb9a1567d9-istiod-ca-cert\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-n5hzj\" (UID: \"28b34a7e-4765-46d0-b412-74eb9a1567d9\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-n5hzj" Oct 30 22:16:50.132870 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:50.132851 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28b34a7e-4765-46d0-b412-74eb9a1567d9-istio-envoy\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-n5hzj\" (UID: \"28b34a7e-4765-46d0-b412-74eb9a1567d9\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-n5hzj" Oct 30 22:16:50.133105 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:50.133087 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28b34a7e-4765-46d0-b412-74eb9a1567d9-istio-podinfo\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-n5hzj\" (UID: \"28b34a7e-4765-46d0-b412-74eb9a1567d9\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-n5hzj" Oct 30 22:16:50.144619 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:50.144592 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28b34a7e-4765-46d0-b412-74eb9a1567d9-istio-token\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-n5hzj\" (UID: \"28b34a7e-4765-46d0-b412-74eb9a1567d9\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-n5hzj" Oct 30 22:16:50.145332 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:50.145309 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4k7v\" (UniqueName: \"kubernetes.io/projected/28b34a7e-4765-46d0-b412-74eb9a1567d9-kube-api-access-k4k7v\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-n5hzj\" (UID: \"28b34a7e-4765-46d0-b412-74eb9a1567d9\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-n5hzj" Oct 30 22:16:50.318699 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:50.318663 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-n5hzj" Oct 30 22:16:50.614375 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:50.614341 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-n5hzj"] Oct 30 22:16:50.615364 ip-10-0-137-181 kubenswrapper[6720]: W1030 22:16:50.615338 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28b34a7e_4765_46d0_b412_74eb9a1567d9.slice/crio-1c3c7d57beb46ceeca529232994f7f8e4282997238ca5dea897b086fd0fb0fa9 WatchSource:0}: Error finding container 1c3c7d57beb46ceeca529232994f7f8e4282997238ca5dea897b086fd0fb0fa9: Status 404 returned error can't find the container with id 1c3c7d57beb46ceeca529232994f7f8e4282997238ca5dea897b086fd0fb0fa9 Oct 30 22:16:50.618773 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:50.618739 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:16:50.618877 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:50.618820 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:16:50.618877 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:50.618868 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:16:50.739483 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:50.737017 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:51.237264 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:51.237218 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-n5hzj" event={"ID":"28b34a7e-4765-46d0-b412-74eb9a1567d9","Type":"ContainerStarted","Data":"2c26c70d220b333ea4dbba0c8cc1cbe4828487c2e752f86fa8d5090f6b391bf1"} Oct 30 22:16:51.237735 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:51.237275 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-n5hzj" event={"ID":"28b34a7e-4765-46d0-b412-74eb9a1567d9","Type":"ContainerStarted","Data":"1c3c7d57beb46ceeca529232994f7f8e4282997238ca5dea897b086fd0fb0fa9"} Oct 30 22:16:51.275021 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:51.274950 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-n5hzj" podStartSLOduration=2.274930389 podStartE2EDuration="2.274930389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:51.273957791 +0000 UTC m=+2177.444922975" watchObservedRunningTime="2026-10-30 22:16:51.274930389 +0000 UTC m=+2177.445895572" Oct 30 22:16:51.319632 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:51.319597 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-n5hzj" Oct 30 22:16:51.321359 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:51.321318 6720 patch_prober.go:29] interesting pod/e2e-extmod-a-a39158-openshift-default-575c648d9-n5hzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 30 22:16:51.321524 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:51.321419 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-n5hzj" podUID="28b34a7e-4765-46d0-b412-74eb9a1567d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 30 22:16:51.590918 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:51.590834 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:52.319305 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:52.319266 6720 patch_prober.go:29] interesting pod/e2e-extmod-a-a39158-openshift-default-575c648d9-n5hzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 30 22:16:52.319735 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:52.319341 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-n5hzj" podUID="28b34a7e-4765-46d0-b412-74eb9a1567d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 30 22:16:52.529823 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:52.529773 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:53.319864 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:53.319819 6720 patch_prober.go:29] interesting pod/e2e-extmod-a-a39158-openshift-default-575c648d9-n5hzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 30 22:16:53.320357 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:53.319884 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-n5hzj" podUID="28b34a7e-4765-46d0-b412-74eb9a1567d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 30 22:16:53.339308 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:53.339274 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:54.042367 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:54.042332 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:54.319657 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:54.319570 6720 patch_prober.go:29] interesting pod/e2e-extmod-a-a39158-openshift-default-575c648d9-n5hzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 30 22:16:54.319657 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:54.319627 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-n5hzj" podUID="28b34a7e-4765-46d0-b412-74eb9a1567d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 30 22:16:54.853408 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:54.844329 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:55.319034 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:55.318951 6720 patch_prober.go:29] interesting pod/e2e-extmod-a-a39158-openshift-default-575c648d9-n5hzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 30 22:16:55.319188 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:55.319043 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-n5hzj" podUID="28b34a7e-4765-46d0-b412-74eb9a1567d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 30 22:16:55.778413 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:55.777706 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:56.319479 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:56.319430 6720 patch_prober.go:29] interesting pod/e2e-extmod-a-a39158-openshift-default-575c648d9-n5hzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 30 22:16:56.320016 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:56.319513 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-n5hzj" podUID="28b34a7e-4765-46d0-b412-74eb9a1567d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 30 22:16:56.649155 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:56.649101 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:57.312122 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:57.311832 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:57.352489 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:57.352461 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-n5hzj" Oct 30 22:16:57.353017 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:57.352802 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-n5hzj" Oct 30 22:16:57.354512 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:57.354488 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-n5hzj" Oct 30 22:16:58.052092 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:58.052054 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:58.989244 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:58.989206 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:16:59.697542 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:16:59.696492 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:00.344416 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:00.343138 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:00.951922 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:00.951887 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:01.675150 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:01.675106 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:02.347182 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:02.347132 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:03.019049 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:03.019011 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:03.720942 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:03.720860 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:04.031263 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:04.031171 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-n4hgm"] Oct 30 22:17:04.031769 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:04.031437 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-n4hgm" podUID="2fa901df-6086-43c3-8f7e-a7a633607486" containerName="istio-proxy" containerID="cri-o://b98c418c3157438e619241611a2e22a614b3a8eb620d9a4f1227ba52f60b3b6b" gracePeriod=30 Oct 30 22:17:04.440289 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:04.440250 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:04.991075 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:04.991035 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-wc924"] Oct 30 22:17:04.995896 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:04.995872 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-wc924" Oct 30 22:17:05.010988 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:05.010959 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-wc924"] Oct 30 22:17:05.089857 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:05.089822 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0051f207-fce7-4aed-bcc3-a5aa69bb9a3c-istio-token\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-wc924\" (UID: \"0051f207-fce7-4aed-bcc3-a5aa69bb9a3c\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-wc924" Oct 30 22:17:05.090278 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:05.090006 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0051f207-fce7-4aed-bcc3-a5aa69bb9a3c-workload-socket\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-wc924\" (UID: \"0051f207-fce7-4aed-bcc3-a5aa69bb9a3c\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-wc924" Oct 30 22:17:05.090278 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:05.090034 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0051f207-fce7-4aed-bcc3-a5aa69bb9a3c-credential-socket\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-wc924\" (UID: \"0051f207-fce7-4aed-bcc3-a5aa69bb9a3c\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-wc924" Oct 30 22:17:05.090278 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:05.090124 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0051f207-fce7-4aed-bcc3-a5aa69bb9a3c-istio-podinfo\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-wc924\" (UID: \"0051f207-fce7-4aed-bcc3-a5aa69bb9a3c\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-wc924" Oct 30 22:17:05.090278 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:05.090263 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0051f207-fce7-4aed-bcc3-a5aa69bb9a3c-istiod-ca-cert\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-wc924\" (UID: \"0051f207-fce7-4aed-bcc3-a5aa69bb9a3c\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-wc924" Oct 30 22:17:05.090483 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:05.090312 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0051f207-fce7-4aed-bcc3-a5aa69bb9a3c-workload-certs\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-wc924\" (UID: \"0051f207-fce7-4aed-bcc3-a5aa69bb9a3c\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-wc924" Oct 30 22:17:05.090483 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:05.090472 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0051f207-fce7-4aed-bcc3-a5aa69bb9a3c-istio-envoy\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-wc924\" (UID: \"0051f207-fce7-4aed-bcc3-a5aa69bb9a3c\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-wc924" Oct 30 22:17:05.090596 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:05.090578 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0051f207-fce7-4aed-bcc3-a5aa69bb9a3c-istio-data\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-wc924\" (UID: \"0051f207-fce7-4aed-bcc3-a5aa69bb9a3c\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-wc924" Oct 30 22:17:05.090641 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:05.090609 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8svxh\" (UniqueName: \"kubernetes.io/projected/0051f207-fce7-4aed-bcc3-a5aa69bb9a3c-kube-api-access-8svxh\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-wc924\" (UID: \"0051f207-fce7-4aed-bcc3-a5aa69bb9a3c\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-wc924" Oct 30 22:17:05.191671 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:05.191638 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0051f207-fce7-4aed-bcc3-a5aa69bb9a3c-istio-token\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-wc924\" (UID: \"0051f207-fce7-4aed-bcc3-a5aa69bb9a3c\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-wc924" Oct 30 22:17:05.191864 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:05.191712 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0051f207-fce7-4aed-bcc3-a5aa69bb9a3c-workload-socket\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-wc924\" (UID: \"0051f207-fce7-4aed-bcc3-a5aa69bb9a3c\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-wc924" Oct 30 22:17:05.191864 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:05.191737 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0051f207-fce7-4aed-bcc3-a5aa69bb9a3c-credential-socket\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-wc924\" (UID: \"0051f207-fce7-4aed-bcc3-a5aa69bb9a3c\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-wc924" Oct 30 22:17:05.191864 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:05.191772 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0051f207-fce7-4aed-bcc3-a5aa69bb9a3c-istio-podinfo\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-wc924\" (UID: \"0051f207-fce7-4aed-bcc3-a5aa69bb9a3c\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-wc924" Oct 30 22:17:05.192043 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:05.191887 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0051f207-fce7-4aed-bcc3-a5aa69bb9a3c-istiod-ca-cert\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-wc924\" (UID: \"0051f207-fce7-4aed-bcc3-a5aa69bb9a3c\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-wc924" Oct 30 22:17:05.192043 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:05.191922 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0051f207-fce7-4aed-bcc3-a5aa69bb9a3c-workload-certs\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-wc924\" (UID: \"0051f207-fce7-4aed-bcc3-a5aa69bb9a3c\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-wc924" Oct 30 22:17:05.192043 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:05.191979 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0051f207-fce7-4aed-bcc3-a5aa69bb9a3c-istio-envoy\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-wc924\" (UID: \"0051f207-fce7-4aed-bcc3-a5aa69bb9a3c\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-wc924" Oct 30 22:17:05.192207 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:05.192079 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0051f207-fce7-4aed-bcc3-a5aa69bb9a3c-istio-data\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-wc924\" (UID: \"0051f207-fce7-4aed-bcc3-a5aa69bb9a3c\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-wc924" Oct 30 22:17:05.192207 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:05.192110 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8svxh\" (UniqueName: \"kubernetes.io/projected/0051f207-fce7-4aed-bcc3-a5aa69bb9a3c-kube-api-access-8svxh\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-wc924\" (UID: \"0051f207-fce7-4aed-bcc3-a5aa69bb9a3c\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-wc924" Oct 30 22:17:05.192321 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:05.192226 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0051f207-fce7-4aed-bcc3-a5aa69bb9a3c-workload-socket\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-wc924\" (UID: \"0051f207-fce7-4aed-bcc3-a5aa69bb9a3c\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-wc924" Oct 30 22:17:05.192321 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:05.192269 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0051f207-fce7-4aed-bcc3-a5aa69bb9a3c-credential-socket\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-wc924\" (UID: \"0051f207-fce7-4aed-bcc3-a5aa69bb9a3c\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-wc924" Oct 30 22:17:05.192723 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:05.192651 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0051f207-fce7-4aed-bcc3-a5aa69bb9a3c-istio-data\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-wc924\" (UID: \"0051f207-fce7-4aed-bcc3-a5aa69bb9a3c\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-wc924" Oct 30 22:17:05.192838 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:05.192732 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0051f207-fce7-4aed-bcc3-a5aa69bb9a3c-istiod-ca-cert\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-wc924\" (UID: \"0051f207-fce7-4aed-bcc3-a5aa69bb9a3c\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-wc924" Oct 30 22:17:05.192907 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:05.192849 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0051f207-fce7-4aed-bcc3-a5aa69bb9a3c-workload-certs\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-wc924\" (UID: \"0051f207-fce7-4aed-bcc3-a5aa69bb9a3c\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-wc924" Oct 30 22:17:05.194874 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:05.194843 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0051f207-fce7-4aed-bcc3-a5aa69bb9a3c-istio-podinfo\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-wc924\" (UID: \"0051f207-fce7-4aed-bcc3-a5aa69bb9a3c\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-wc924" Oct 30 22:17:05.195009 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:05.194988 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0051f207-fce7-4aed-bcc3-a5aa69bb9a3c-istio-envoy\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-wc924\" (UID: \"0051f207-fce7-4aed-bcc3-a5aa69bb9a3c\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-wc924" Oct 30 22:17:05.209694 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:05.209661 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0051f207-fce7-4aed-bcc3-a5aa69bb9a3c-istio-token\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-wc924\" (UID: \"0051f207-fce7-4aed-bcc3-a5aa69bb9a3c\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-wc924" Oct 30 22:17:05.209908 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:05.209890 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8svxh\" (UniqueName: \"kubernetes.io/projected/0051f207-fce7-4aed-bcc3-a5aa69bb9a3c-kube-api-access-8svxh\") pod \"e2e-extmod-a-a39158-openshift-default-575c648d9-wc924\" (UID: \"0051f207-fce7-4aed-bcc3-a5aa69bb9a3c\") " pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-wc924" Oct 30 22:17:05.329323 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:05.329237 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-wc924" Oct 30 22:17:05.335139 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:05.335059 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:05.659182 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:05.659155 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-wc924"] Oct 30 22:17:05.659956 ip-10-0-137-181 kubenswrapper[6720]: W1030 22:17:05.659928 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0051f207_fce7_4aed_bcc3_a5aa69bb9a3c.slice/crio-eeec6c394d2f953dd5ce92b294c3e39989e06b8de2c7dba8e6396617f53cb123 WatchSource:0}: Error finding container eeec6c394d2f953dd5ce92b294c3e39989e06b8de2c7dba8e6396617f53cb123: Status 404 returned error can't find the container with id eeec6c394d2f953dd5ce92b294c3e39989e06b8de2c7dba8e6396617f53cb123 Oct 30 22:17:05.662806 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:05.662772 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:17:05.662880 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:05.662838 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:17:05.662880 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:05.662870 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:17:06.035828 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:06.035694 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:06.422915 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:06.422749 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-wc924" event={"ID":"0051f207-fce7-4aed-bcc3-a5aa69bb9a3c","Type":"ContainerStarted","Data":"76650545cd685ef5bfdeebaaa3a26dbf09d04f9afff648f4fa051bd6185ecf4e"} Oct 30 22:17:06.422915 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:06.422829 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-wc924" event={"ID":"0051f207-fce7-4aed-bcc3-a5aa69bb9a3c","Type":"ContainerStarted","Data":"eeec6c394d2f953dd5ce92b294c3e39989e06b8de2c7dba8e6396617f53cb123"} Oct 30 22:17:06.476892 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:06.476830 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-wc924" podStartSLOduration=2.476810581 podStartE2EDuration="2.476810581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:06.472336715 +0000 UTC m=+2192.643301897" watchObservedRunningTime="2026-10-30 22:17:06.476810581 +0000 UTC m=+2192.647775763" Oct 30 22:17:06.870407 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:06.863832 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:07.329760 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:07.329678 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-wc924" Oct 30 22:17:07.331247 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:07.331219 6720 patch_prober.go:29] interesting pod/e2e-extmod-a-a39158-openshift-default-575c648d9-wc924 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 30 22:17:07.331439 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:07.331273 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-wc924" podUID="0051f207-fce7-4aed-bcc3-a5aa69bb9a3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 30 22:17:07.641628 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:07.641590 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:08.329930 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:08.329888 6720 patch_prober.go:29] interesting pod/e2e-extmod-a-a39158-openshift-default-575c648d9-wc924 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 30 22:17:08.330083 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:08.329964 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-wc924" podUID="0051f207-fce7-4aed-bcc3-a5aa69bb9a3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 30 22:17:08.439769 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:08.439598 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:09.243218 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:09.242025 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:09.330343 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:09.330308 6720 patch_prober.go:29] interesting pod/e2e-extmod-a-a39158-openshift-default-575c648d9-wc924 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 30 22:17:09.330564 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:09.330365 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-wc924" podUID="0051f207-fce7-4aed-bcc3-a5aa69bb9a3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 30 22:17:09.431863 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:09.431838 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-n4hgm" Oct 30 22:17:09.487560 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:09.487527 6720 generic.go:356] "Generic (PLEG): container finished" podID="2fa901df-6086-43c3-8f7e-a7a633607486" containerID="b98c418c3157438e619241611a2e22a614b3a8eb620d9a4f1227ba52f60b3b6b" exitCode=0 Oct 30 22:17:09.487882 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:09.487857 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-n4hgm" event={"ID":"2fa901df-6086-43c3-8f7e-a7a633607486","Type":"ContainerDied","Data":"b98c418c3157438e619241611a2e22a614b3a8eb620d9a4f1227ba52f60b3b6b"} Oct 30 22:17:09.488158 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:09.488145 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-n4hgm" Oct 30 22:17:09.488396 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:09.488213 6720 scope.go:122] "RemoveContainer" containerID="b98c418c3157438e619241611a2e22a614b3a8eb620d9a4f1227ba52f60b3b6b" Oct 30 22:17:09.488522 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:09.488503 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-n4hgm" event={"ID":"2fa901df-6086-43c3-8f7e-a7a633607486","Type":"ContainerDied","Data":"92015ddc3028564eb797fd02e1ba42a3b25f545a1fbab7df289575334f162528"} Oct 30 22:17:09.503602 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:09.503547 6720 scope.go:122] "RemoveContainer" containerID="b98c418c3157438e619241611a2e22a614b3a8eb620d9a4f1227ba52f60b3b6b" Oct 30 22:17:09.504243 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:17:09.504211 6720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b98c418c3157438e619241611a2e22a614b3a8eb620d9a4f1227ba52f60b3b6b\": container with ID starting with b98c418c3157438e619241611a2e22a614b3a8eb620d9a4f1227ba52f60b3b6b not found: ID does not exist" containerID="b98c418c3157438e619241611a2e22a614b3a8eb620d9a4f1227ba52f60b3b6b" Oct 30 22:17:09.504341 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:09.504257 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b98c418c3157438e619241611a2e22a614b3a8eb620d9a4f1227ba52f60b3b6b"} err="failed to get container status \"b98c418c3157438e619241611a2e22a614b3a8eb620d9a4f1227ba52f60b3b6b\": rpc error: code = NotFound desc = could not find container \"b98c418c3157438e619241611a2e22a614b3a8eb620d9a4f1227ba52f60b3b6b\": container with ID starting with b98c418c3157438e619241611a2e22a614b3a8eb620d9a4f1227ba52f60b3b6b not found: ID does not exist" Oct 30 22:17:09.604554 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:09.604516 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fa901df-6086-43c3-8f7e-a7a633607486-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fa901df-6086-43c3-8f7e-a7a633607486-workload-socket\") pod \"2fa901df-6086-43c3-8f7e-a7a633607486\" (UID: \"2fa901df-6086-43c3-8f7e-a7a633607486\") " Oct 30 22:17:09.604857 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:09.604563 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fa901df-6086-43c3-8f7e-a7a633607486-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fa901df-6086-43c3-8f7e-a7a633607486-credential-socket\") pod \"2fa901df-6086-43c3-8f7e-a7a633607486\" (UID: \"2fa901df-6086-43c3-8f7e-a7a633607486\") " Oct 30 22:17:09.604857 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:09.604646 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fa901df-6086-43c3-8f7e-a7a633607486-kube-api-access-c8hbn\" (UniqueName: \"kubernetes.io/projected/2fa901df-6086-43c3-8f7e-a7a633607486-kube-api-access-c8hbn\") pod \"2fa901df-6086-43c3-8f7e-a7a633607486\" (UID: \"2fa901df-6086-43c3-8f7e-a7a633607486\") " Oct 30 22:17:09.604857 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:09.604746 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fa901df-6086-43c3-8f7e-a7a633607486-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fa901df-6086-43c3-8f7e-a7a633607486-istiod-ca-cert\") pod \"2fa901df-6086-43c3-8f7e-a7a633607486\" (UID: \"2fa901df-6086-43c3-8f7e-a7a633607486\") " Oct 30 22:17:09.604857 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:09.604822 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fa901df-6086-43c3-8f7e-a7a633607486-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fa901df-6086-43c3-8f7e-a7a633607486-istio-data\") pod \"2fa901df-6086-43c3-8f7e-a7a633607486\" (UID: \"2fa901df-6086-43c3-8f7e-a7a633607486\") " Oct 30 22:17:09.605220 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:09.604882 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fa901df-6086-43c3-8f7e-a7a633607486-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fa901df-6086-43c3-8f7e-a7a633607486-workload-certs\") pod \"2fa901df-6086-43c3-8f7e-a7a633607486\" (UID: \"2fa901df-6086-43c3-8f7e-a7a633607486\") " Oct 30 22:17:09.605220 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:09.604922 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fa901df-6086-43c3-8f7e-a7a633607486-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fa901df-6086-43c3-8f7e-a7a633607486-istio-envoy\") pod \"2fa901df-6086-43c3-8f7e-a7a633607486\" (UID: \"2fa901df-6086-43c3-8f7e-a7a633607486\") " Oct 30 22:17:09.605220 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:09.604920 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fa901df-6086-43c3-8f7e-a7a633607486-credential-socket" pod "2fa901df-6086-43c3-8f7e-a7a633607486" (UID: "2fa901df-6086-43c3-8f7e-a7a633607486"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:09.605220 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:09.604931 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fa901df-6086-43c3-8f7e-a7a633607486-workload-socket" pod "2fa901df-6086-43c3-8f7e-a7a633607486" (UID: "2fa901df-6086-43c3-8f7e-a7a633607486"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:09.605220 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:09.604995 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2fa901df-6086-43c3-8f7e-a7a633607486-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fa901df-6086-43c3-8f7e-a7a633607486-istio-podinfo\") pod \"2fa901df-6086-43c3-8f7e-a7a633607486\" (UID: \"2fa901df-6086-43c3-8f7e-a7a633607486\") " Oct 30 22:17:09.605220 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:09.605139 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fa901df-6086-43c3-8f7e-a7a633607486-istio-token\" (UniqueName: \"kubernetes.io/projected/2fa901df-6086-43c3-8f7e-a7a633607486-istio-token\") pod \"2fa901df-6086-43c3-8f7e-a7a633607486\" (UID: \"2fa901df-6086-43c3-8f7e-a7a633607486\") " Oct 30 22:17:09.605220 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:09.605171 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fa901df-6086-43c3-8f7e-a7a633607486-workload-certs" pod "2fa901df-6086-43c3-8f7e-a7a633607486" (UID: "2fa901df-6086-43c3-8f7e-a7a633607486"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:09.605220 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:09.605193 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fa901df-6086-43c3-8f7e-a7a633607486-istio-data" pod "2fa901df-6086-43c3-8f7e-a7a633607486" (UID: "2fa901df-6086-43c3-8f7e-a7a633607486"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:09.605220 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:09.605213 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fa901df-6086-43c3-8f7e-a7a633607486-istiod-ca-cert" pod "2fa901df-6086-43c3-8f7e-a7a633607486" (UID: "2fa901df-6086-43c3-8f7e-a7a633607486"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:17:09.606019 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:09.605991 6720 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fa901df-6086-43c3-8f7e-a7a633607486-workload-certs\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:09.606091 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:09.606023 6720 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fa901df-6086-43c3-8f7e-a7a633607486-workload-socket\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:09.606091 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:09.606039 6720 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2fa901df-6086-43c3-8f7e-a7a633607486-credential-socket\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:09.606091 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:09.606054 6720 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2fa901df-6086-43c3-8f7e-a7a633607486-istiod-ca-cert\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:09.606091 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:09.606069 6720 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2fa901df-6086-43c3-8f7e-a7a633607486-istio-data\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:09.607158 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:09.607137 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fa901df-6086-43c3-8f7e-a7a633607486-kube-api-access-c8hbn" pod "2fa901df-6086-43c3-8f7e-a7a633607486" (UID: "2fa901df-6086-43c3-8f7e-a7a633607486"). InnerVolumeSpecName "kube-api-access-c8hbn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:09.607441 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:09.607419 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2fa901df-6086-43c3-8f7e-a7a633607486-istio-podinfo" pod "2fa901df-6086-43c3-8f7e-a7a633607486" (UID: "2fa901df-6086-43c3-8f7e-a7a633607486"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:17:09.607956 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:09.607933 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fa901df-6086-43c3-8f7e-a7a633607486-istio-token" pod "2fa901df-6086-43c3-8f7e-a7a633607486" (UID: "2fa901df-6086-43c3-8f7e-a7a633607486"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:09.607956 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:09.607933 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fa901df-6086-43c3-8f7e-a7a633607486-istio-envoy" pod "2fa901df-6086-43c3-8f7e-a7a633607486" (UID: "2fa901df-6086-43c3-8f7e-a7a633607486"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:17:09.707869 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:09.707834 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8hbn\" (UniqueName: \"kubernetes.io/projected/2fa901df-6086-43c3-8f7e-a7a633607486-kube-api-access-c8hbn\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:09.708061 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:09.707867 6720 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2fa901df-6086-43c3-8f7e-a7a633607486-istio-envoy\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:09.708061 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:09.707896 6720 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2fa901df-6086-43c3-8f7e-a7a633607486-istio-podinfo\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:09.708061 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:09.707907 6720 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fa901df-6086-43c3-8f7e-a7a633607486-istio-token\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:09.815770 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:09.815736 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-n4hgm"] Oct 30 22:17:09.821616 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:09.821584 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-5313f0-openshift-default-bf8578f7f-n4hgm"] Oct 30 22:17:09.978769 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:09.978688 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fa901df-6086-43c3-8f7e-a7a633607486" path="/var/lib/kubelet/pods/2fa901df-6086-43c3-8f7e-a7a633607486/volumes" Oct 30 22:17:10.044129 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:10.044086 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:10.329926 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:10.329849 6720 patch_prober.go:29] interesting pod/e2e-extmod-a-a39158-openshift-default-575c648d9-wc924 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 30 22:17:10.330396 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:10.330354 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-wc924" podUID="0051f207-fce7-4aed-bcc3-a5aa69bb9a3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 30 22:17:10.405729 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:17:10.405689 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 22:17:10.723349 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:10.723304 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:11.330462 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:11.330423 6720 patch_prober.go:29] interesting pod/e2e-extmod-a-a39158-openshift-default-575c648d9-wc924 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 30 22:17:11.330881 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:11.330470 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-wc924" podUID="0051f207-fce7-4aed-bcc3-a5aa69bb9a3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 30 22:17:11.500409 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:11.500008 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:12.200588 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:12.200549 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:12.335916 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:12.335886 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-wc924" Oct 30 22:17:12.336439 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:12.336233 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-wc924" Oct 30 22:17:12.336954 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:12.336937 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-wc924" Oct 30 22:17:12.853211 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:12.846916 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:13.485051 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:13.483766 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:14.237433 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:14.234916 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:14.917958 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:14.917915 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:15.596281 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:15.596249 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:16.353734 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:16.353558 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:17.112804 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:17.111776 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:17.733824 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:17.733738 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:18.445407 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:18.444972 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:19.211045 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:19.210089 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:20.005551 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:20.005466 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:20.727250 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:20.727208 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:21.412271 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:21.412234 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:22.221467 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:22.218226 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:23.005640 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:23.005593 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:23.826577 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:23.826512 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:24.732415 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:24.731609 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:25.394812 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:25.394455 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:26.109670 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:26.109621 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:26.721740 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:26.721701 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:27.563360 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:27.563313 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:27.886003 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:27.885972 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:28.141916 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:28.141827 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:28.484663 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:28.484585 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:28.897424 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:28.895883 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:29.119306 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:29.119265 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:29.269287 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:29.269188 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:29.537507 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:29.527723 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:30.243707 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:30.243654 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:30.941531 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:30.941490 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:31.669410 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:31.659209 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:32.372429 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:32.372340 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:33.060164 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:33.060117 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:33.924349 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:33.921979 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:34.736419 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:34.735878 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:35.514158 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:35.511656 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:36.109248 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:36.109214 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:36.310413 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:36.307894 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:36.545447 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:36.545321 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:36.700288 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:36.700239 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:36.967763 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:36.967727 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:37.485403 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:37.485347 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:38.139322 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:38.139235 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:38.319287 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:38.319239 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:39.107985 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:39.107950 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:39.882222 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:39.882176 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:40.405759 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:17:40.405709 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 22:17:40.880929 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:40.880895 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:41.624024 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:41.623965 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:42.408415 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:42.407599 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:43.029584 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:43.014008 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:43.284282 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:43.284179 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:43.483417 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:43.474285 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:43.656345 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:43.656285 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:43.868216 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:43.868177 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:44.071315 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:44.071225 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:44.356208 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:44.356183 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:44.539233 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:44.539200 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:44.849480 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:44.849373 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:45.028410 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:45.028362 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:45.526730 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:45.526696 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:45.687252 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:45.687218 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:45.833289 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:45.833175 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:46.041210 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:46.041156 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:46.490378 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:46.490345 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:46.711648 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:46.711575 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:46.873464 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:46.873272 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:47.270696 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:47.270617 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:47.442593 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:47.442553 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:48.079778 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:48.079738 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:48.283079 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:48.283025 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:48.450323 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:48.450288 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:49.422606 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:49.422568 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:50.140996 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:50.140957 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:50.836605 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:50.836562 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:51.542537 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:51.542500 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:51.741191 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:51.737326 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:51.901617 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:51.901582 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:52.216421 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:52.216261 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:52.868109 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:52.868059 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:53.055483 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:53.047416 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:53.430089 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:53.430057 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:54.227202 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:54.227132 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:54.968500 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:54.968361 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:55.670235 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:55.669729 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:56.414368 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:56.414309 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:57.047202 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:57.047162 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:57.822207 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:57.822168 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:58.428041 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:58.428000 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:59.023736 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:59.023702 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:17:59.923486 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:17:59.919450 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:00.103357 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:00.103316 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:00.252727 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:00.252629 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:00.394635 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:00.394608 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:00.554460 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:00.554365 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:00.833995 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:00.833882 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:01.150964 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:01.150915 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:01.365048 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:01.365013 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:01.545343 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:01.545245 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:01.870045 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:01.870008 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:02.041065 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:02.041025 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:02.368544 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:02.368511 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:03.116212 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:03.116176 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:03.732351 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:03.732312 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:04.601655 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:04.600356 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:05.220411 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:05.214192 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:05.880671 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:05.880630 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:06.636415 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:06.634502 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:07.432298 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:07.432263 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:08.184055 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:08.184022 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:08.854340 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:08.854279 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:09.109329 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:09.109224 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:09.294693 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:09.294664 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:09.496328 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:09.496249 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:09.722112 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:09.722082 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:10.404698 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:18:10.404663 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 22:18:10.445886 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:10.445805 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:11.247103 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:11.247061 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:11.949834 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:11.949293 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:12.661411 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:12.660681 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:13.774345 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:13.774260 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:14.026834 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:14.026755 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:14.541533 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:14.541487 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:15.258452 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:15.254342 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:16.094299 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:16.094258 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:17.012189 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:17.012140 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:17.174832 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:17.174788 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:17.383411 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:17.380427 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:17.564928 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:17.564880 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:18.125455 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:18.125424 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:18.286873 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:18.286840 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:18.797635 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:18.797598 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:19.875119 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:19.875081 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:20.100414 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:20.086246 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:20.457414 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:20.453188 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:21.955427 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:21.955372 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:22.346665 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:22.346560 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:22.525716 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:22.525680 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:23.018409 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:23.010852 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:23.979884 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:23.979794 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:25.096954 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:25.096916 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:25.328952 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:25.328907 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:26.432875 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:26.432815 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:27.216410 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:27.215448 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:27.825958 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:27.825926 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:28.128115 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:28.128081 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:28.336499 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:28.336470 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:28.525066 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:28.524984 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:29.075581 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:29.075545 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:29.333056 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:29.332970 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:29.807079 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:29.807036 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:29.987947 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:29.987894 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:30.661357 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:30.661308 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:31.543911 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:31.543867 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:32.250628 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:32.250592 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:33.005166 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:33.005130 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:33.640470 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:33.640432 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:34.320895 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:34.320860 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:35.090977 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:35.090939 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:35.751605 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:35.750551 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:36.632594 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:36.630974 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:37.329263 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:37.329198 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:37.997993 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:37.997962 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:38.725503 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:38.724431 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:39.426996 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:39.426960 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:40.142771 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:40.142722 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:40.404959 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:18:40.404851 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 22:18:40.967501 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:40.965496 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:41.794355 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:41.794319 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:42.459489 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:42.459448 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:42.468186 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:42.468158 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grgv"] Oct 30 22:18:42.469148 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:42.469122 6720 cpu_manager.go:414] "containerMap: removing container" podUID="2fa901df-6086-43c3-8f7e-a7a633607486" containerName="istio-proxy" Oct 30 22:18:42.469148 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:42.469146 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fa901df-6086-43c3-8f7e-a7a633607486" containerName="istio-proxy" Oct 30 22:18:42.469432 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:42.469412 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fa901df-6086-43c3-8f7e-a7a633607486" containerName="istio-proxy" Oct 30 22:18:42.474151 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:42.474129 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grgv" Oct 30 22:18:42.495668 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:42.495640 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grgv"] Oct 30 22:18:42.509437 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:42.509404 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9200859d-b207-4764-a75c-270ba29b4155-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8grgv\" (UID: \"9200859d-b207-4764-a75c-270ba29b4155\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grgv" Oct 30 22:18:42.509613 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:42.509529 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9200859d-b207-4764-a75c-270ba29b4155-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8grgv\" (UID: \"9200859d-b207-4764-a75c-270ba29b4155\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grgv" Oct 30 22:18:42.509613 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:42.509588 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9200859d-b207-4764-a75c-270ba29b4155-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8grgv\" (UID: \"9200859d-b207-4764-a75c-270ba29b4155\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grgv" Oct 30 22:18:42.509740 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:42.509723 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9200859d-b207-4764-a75c-270ba29b4155-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8grgv\" (UID: \"9200859d-b207-4764-a75c-270ba29b4155\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grgv" Oct 30 22:18:42.509823 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:42.509749 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9200859d-b207-4764-a75c-270ba29b4155-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8grgv\" (UID: \"9200859d-b207-4764-a75c-270ba29b4155\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grgv" Oct 30 22:18:42.509873 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:42.509856 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9200859d-b207-4764-a75c-270ba29b4155-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8grgv\" (UID: \"9200859d-b207-4764-a75c-270ba29b4155\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grgv" Oct 30 22:18:42.510099 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:42.510075 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl6zv\" (UniqueName: \"kubernetes.io/projected/9200859d-b207-4764-a75c-270ba29b4155-kube-api-access-wl6zv\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8grgv\" (UID: \"9200859d-b207-4764-a75c-270ba29b4155\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grgv" Oct 30 22:18:42.510205 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:42.510189 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9200859d-b207-4764-a75c-270ba29b4155-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8grgv\" (UID: \"9200859d-b207-4764-a75c-270ba29b4155\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grgv" Oct 30 22:18:42.510244 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:42.510232 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9200859d-b207-4764-a75c-270ba29b4155-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8grgv\" (UID: \"9200859d-b207-4764-a75c-270ba29b4155\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grgv" Oct 30 22:18:42.611156 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:42.611117 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9200859d-b207-4764-a75c-270ba29b4155-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8grgv\" (UID: \"9200859d-b207-4764-a75c-270ba29b4155\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grgv" Oct 30 22:18:42.611368 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:42.611170 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9200859d-b207-4764-a75c-270ba29b4155-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8grgv\" (UID: \"9200859d-b207-4764-a75c-270ba29b4155\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grgv" Oct 30 22:18:42.611368 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:42.611220 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9200859d-b207-4764-a75c-270ba29b4155-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8grgv\" (UID: \"9200859d-b207-4764-a75c-270ba29b4155\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grgv" Oct 30 22:18:42.611368 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:42.611236 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9200859d-b207-4764-a75c-270ba29b4155-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8grgv\" (UID: \"9200859d-b207-4764-a75c-270ba29b4155\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grgv" Oct 30 22:18:42.611368 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:42.611289 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9200859d-b207-4764-a75c-270ba29b4155-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8grgv\" (UID: \"9200859d-b207-4764-a75c-270ba29b4155\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grgv" Oct 30 22:18:42.611663 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:42.611413 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl6zv\" (UniqueName: \"kubernetes.io/projected/9200859d-b207-4764-a75c-270ba29b4155-kube-api-access-wl6zv\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8grgv\" (UID: \"9200859d-b207-4764-a75c-270ba29b4155\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grgv" Oct 30 22:18:42.611663 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:42.611487 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9200859d-b207-4764-a75c-270ba29b4155-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8grgv\" (UID: \"9200859d-b207-4764-a75c-270ba29b4155\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grgv" Oct 30 22:18:42.611779 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:42.611662 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9200859d-b207-4764-a75c-270ba29b4155-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8grgv\" (UID: \"9200859d-b207-4764-a75c-270ba29b4155\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grgv" Oct 30 22:18:42.611779 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:42.611716 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9200859d-b207-4764-a75c-270ba29b4155-credential-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8grgv\" (UID: \"9200859d-b207-4764-a75c-270ba29b4155\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grgv" Oct 30 22:18:42.611779 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:42.611741 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9200859d-b207-4764-a75c-270ba29b4155-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8grgv\" (UID: \"9200859d-b207-4764-a75c-270ba29b4155\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grgv" Oct 30 22:18:42.611779 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:42.611660 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9200859d-b207-4764-a75c-270ba29b4155-istio-data\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8grgv\" (UID: \"9200859d-b207-4764-a75c-270ba29b4155\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grgv" Oct 30 22:18:42.611968 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:42.611932 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9200859d-b207-4764-a75c-270ba29b4155-workload-certs\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8grgv\" (UID: \"9200859d-b207-4764-a75c-270ba29b4155\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grgv" Oct 30 22:18:42.612113 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:42.612092 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9200859d-b207-4764-a75c-270ba29b4155-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8grgv\" (UID: \"9200859d-b207-4764-a75c-270ba29b4155\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grgv" Oct 30 22:18:42.612334 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:42.612313 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9200859d-b207-4764-a75c-270ba29b4155-workload-socket\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8grgv\" (UID: \"9200859d-b207-4764-a75c-270ba29b4155\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grgv" Oct 30 22:18:42.613804 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:42.613781 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9200859d-b207-4764-a75c-270ba29b4155-istio-envoy\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8grgv\" (UID: \"9200859d-b207-4764-a75c-270ba29b4155\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grgv" Oct 30 22:18:42.614016 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:42.613994 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9200859d-b207-4764-a75c-270ba29b4155-istio-podinfo\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8grgv\" (UID: \"9200859d-b207-4764-a75c-270ba29b4155\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grgv" Oct 30 22:18:42.624277 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:42.624249 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9200859d-b207-4764-a75c-270ba29b4155-istio-token\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8grgv\" (UID: \"9200859d-b207-4764-a75c-270ba29b4155\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grgv" Oct 30 22:18:42.624590 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:42.624570 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl6zv\" (UniqueName: \"kubernetes.io/projected/9200859d-b207-4764-a75c-270ba29b4155-kube-api-access-wl6zv\") pod \"maas-default-gateway-openshift-default-85544d4cd7-8grgv\" (UID: \"9200859d-b207-4764-a75c-270ba29b4155\") " pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grgv" Oct 30 22:18:42.797586 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:42.797488 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grgv" Oct 30 22:18:43.097762 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:43.097730 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grgv"] Oct 30 22:18:43.099438 ip-10-0-137-181 kubenswrapper[6720]: W1030 22:18:43.099408 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9200859d_b207_4764_a75c_270ba29b4155.slice/crio-9988e7971cd25c4c578cf2f72aa8fb4d74f90daa9bc013228150be0a65c99e11 WatchSource:0}: Error finding container 9988e7971cd25c4c578cf2f72aa8fb4d74f90daa9bc013228150be0a65c99e11: Status 404 returned error can't find the container with id 9988e7971cd25c4c578cf2f72aa8fb4d74f90daa9bc013228150be0a65c99e11 Oct 30 22:18:43.104063 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:43.103589 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:18:43.104063 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:43.103662 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:18:43.104063 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:43.103698 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:18:43.232408 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:43.228767 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:43.570621 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:43.570585 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grgv" event={"ID":"9200859d-b207-4764-a75c-270ba29b4155","Type":"ContainerStarted","Data":"9a24b720f4b35ea65db5cf9b4d366e1758d35bb5ba5fb4bc716d0e760d1f9dde"} Oct 30 22:18:43.571103 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:43.570642 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grgv" event={"ID":"9200859d-b207-4764-a75c-270ba29b4155","Type":"ContainerStarted","Data":"9988e7971cd25c4c578cf2f72aa8fb4d74f90daa9bc013228150be0a65c99e11"} Oct 30 22:18:43.599487 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:43.599430 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grgv" podStartSLOduration=1.59941433 podStartE2EDuration="1.59941433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:43.596285875 +0000 UTC m=+2289.767251058" watchObservedRunningTime="2026-10-30 22:18:43.59941433 +0000 UTC m=+2289.770379513" Oct 30 22:18:43.798064 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:43.798033 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grgv" Oct 30 22:18:43.982413 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:43.969235 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:44.728947 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:44.725267 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:44.799326 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:44.799291 6720 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-8grgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Oct 30 22:18:44.799938 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:44.799644 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grgv" podUID="9200859d-b207-4764-a75c-270ba29b4155" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Oct 30 22:18:45.615413 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:45.611652 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:45.798785 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:45.798750 6720 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-8grgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Oct 30 22:18:45.799288 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:45.798814 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grgv" podUID="9200859d-b207-4764-a75c-270ba29b4155" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Oct 30 22:18:46.327523 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:46.323964 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:46.797909 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:46.797877 6720 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-8grgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Oct 30 22:18:46.798133 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:46.797947 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grgv" podUID="9200859d-b207-4764-a75c-270ba29b4155" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Oct 30 22:18:47.342934 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:47.342876 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:47.798099 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:47.798060 6720 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-8grgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Oct 30 22:18:47.798299 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:47.798131 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grgv" podUID="9200859d-b207-4764-a75c-270ba29b4155" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Oct 30 22:18:48.151810 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:48.151777 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:48.803421 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:48.803357 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:48.805834 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:48.804049 6720 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-8grgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Oct 30 22:18:48.805834 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:48.804130 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grgv" podUID="9200859d-b207-4764-a75c-270ba29b4155" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Oct 30 22:18:49.655906 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:49.655872 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:49.798506 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:49.798470 6720 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-8grgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Oct 30 22:18:49.798689 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:49.798528 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grgv" podUID="9200859d-b207-4764-a75c-270ba29b4155" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Oct 30 22:18:50.309408 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:50.308482 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:50.798238 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:50.798196 6720 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-85544d4cd7-8grgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Oct 30 22:18:50.798454 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:50.798251 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grgv" podUID="9200859d-b207-4764-a75c-270ba29b4155" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Oct 30 22:18:51.016224 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:51.016185 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:51.653329 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:51.653273 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:52.297880 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:52.297845 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grgv" Oct 30 22:18:52.298501 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:52.298477 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grgv" Oct 30 22:18:52.299628 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:52.299606 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grgv" Oct 30 22:18:52.350539 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:52.350503 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:53.193608 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:53.193527 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:53.847603 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:53.847550 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:54.614205 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:54.614173 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:55.435453 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:55.435247 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:56.371572 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:56.370875 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:57.353657 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:57.353611 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:58.415269 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:58.415215 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:59.068796 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:59.068758 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:18:59.848845 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:18:59.848810 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:00.511411 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:00.505856 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:01.313003 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:01.312962 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:01.978232 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:01.978112 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:02.750410 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:02.749018 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:03.542364 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:03.542323 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:04.181602 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:04.181547 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:04.834739 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:04.834675 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:05.737421 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:05.737370 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:06.677667 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:06.677632 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:07.400197 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:07.400154 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:08.145412 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:08.143905 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:08.865408 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:08.860036 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:09.642410 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:09.641987 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:10.391215 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:10.391034 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:10.465098 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:19:10.465061 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 22:19:11.231947 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:11.231908 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:11.956219 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:11.956182 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:12.708695 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:12.708482 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:13.881942 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:13.881897 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:14.175404 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:14.175299 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:14.920270 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:14.920220 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:15.631773 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:15.631736 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:16.348618 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:16.348585 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:17.034591 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:17.034478 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:17.759160 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:17.759079 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:18.415978 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:18.415927 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:19.048752 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:19.048714 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:19.854412 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:19.852170 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:20.622406 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:20.619152 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:21.326603 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:21.324072 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:22.049525 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:22.049455 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:22.795455 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:22.795359 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:23.431477 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:23.431438 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:24.159027 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:24.158991 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:24.837003 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:24.836958 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:25.515087 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:25.515045 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:26.191312 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:26.190662 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:26.918427 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:26.918365 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:27.585199 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:27.585160 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:28.324375 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:28.324337 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:29.151366 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:29.151330 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:29.341124 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:29.341084 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:30.038712 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:30.038678 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:31.080686 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:31.078980 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:31.684410 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:31.683592 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:32.337628 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:32.337595 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:33.051355 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:33.051296 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:33.702449 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:33.699807 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:34.348654 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:34.348534 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:35.161482 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:35.151687 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:35.837412 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:35.823076 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:36.458709 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:36.458433 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:37.241704 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:37.241668 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:37.922706 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:37.922581 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:38.533420 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:38.530186 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:39.357049 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:39.357011 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:40.053379 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:40.053339 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:40.404880 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:19:40.404846 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 22:19:40.822613 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:40.818770 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:41.524487 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:41.524377 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:42.238356 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:42.238320 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-cjmnk"] Oct 30 22:19:42.238934 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:42.238898 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-cjmnk" podUID="5359ba6b-7deb-4055-9069-1e59a9c4e392" containerName="istio-proxy" containerID="cri-o://da85a37615c3f04fa1c21365d780a1fa28e97123dc6dea03c41dffa8a75c4963" gracePeriod=30 Oct 30 22:19:42.413935 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:42.413037 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:42.657413 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:42.656526 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:42.820555 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:42.820512 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:43.257409 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:43.250984 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:43.727918 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:43.727882 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:44.005409 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:44.005295 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:44.180875 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:44.180839 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:45.305557 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:45.305446 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:46.188175 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:46.188136 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:46.826101 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:46.817880 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:47.574638 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:47.574489 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-cjmnk" Oct 30 22:19:47.605424 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:47.600132 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:47.699573 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:47.699535 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5359ba6b-7deb-4055-9069-1e59a9c4e392-istio-token\" (UniqueName: \"kubernetes.io/projected/5359ba6b-7deb-4055-9069-1e59a9c4e392-istio-token\") pod \"5359ba6b-7deb-4055-9069-1e59a9c4e392\" (UID: \"5359ba6b-7deb-4055-9069-1e59a9c4e392\") " Oct 30 22:19:47.699773 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:47.699688 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5359ba6b-7deb-4055-9069-1e59a9c4e392-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5359ba6b-7deb-4055-9069-1e59a9c4e392-workload-certs\") pod \"5359ba6b-7deb-4055-9069-1e59a9c4e392\" (UID: \"5359ba6b-7deb-4055-9069-1e59a9c4e392\") " Oct 30 22:19:47.699851 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:47.699785 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5359ba6b-7deb-4055-9069-1e59a9c4e392-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5359ba6b-7deb-4055-9069-1e59a9c4e392-workload-socket\") pod \"5359ba6b-7deb-4055-9069-1e59a9c4e392\" (UID: \"5359ba6b-7deb-4055-9069-1e59a9c4e392\") " Oct 30 22:19:47.700060 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:47.700005 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5359ba6b-7deb-4055-9069-1e59a9c4e392-workload-certs" pod "5359ba6b-7deb-4055-9069-1e59a9c4e392" (UID: "5359ba6b-7deb-4055-9069-1e59a9c4e392"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:47.700179 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:47.700079 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5359ba6b-7deb-4055-9069-1e59a9c4e392-credential-socket" pod "5359ba6b-7deb-4055-9069-1e59a9c4e392" (UID: "5359ba6b-7deb-4055-9069-1e59a9c4e392"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:47.700179 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:47.700123 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5359ba6b-7deb-4055-9069-1e59a9c4e392-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5359ba6b-7deb-4055-9069-1e59a9c4e392-credential-socket\") pod \"5359ba6b-7deb-4055-9069-1e59a9c4e392\" (UID: \"5359ba6b-7deb-4055-9069-1e59a9c4e392\") " Oct 30 22:19:47.700269 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:47.700209 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5359ba6b-7deb-4055-9069-1e59a9c4e392-kube-api-access-98jht\" (UniqueName: \"kubernetes.io/projected/5359ba6b-7deb-4055-9069-1e59a9c4e392-kube-api-access-98jht\") pod \"5359ba6b-7deb-4055-9069-1e59a9c4e392\" (UID: \"5359ba6b-7deb-4055-9069-1e59a9c4e392\") " Oct 30 22:19:47.700325 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:47.700313 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5359ba6b-7deb-4055-9069-1e59a9c4e392-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5359ba6b-7deb-4055-9069-1e59a9c4e392-istio-data\") pod \"5359ba6b-7deb-4055-9069-1e59a9c4e392\" (UID: \"5359ba6b-7deb-4055-9069-1e59a9c4e392\") " Oct 30 22:19:47.700405 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:47.700363 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5359ba6b-7deb-4055-9069-1e59a9c4e392-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5359ba6b-7deb-4055-9069-1e59a9c4e392-istiod-ca-cert\") pod \"5359ba6b-7deb-4055-9069-1e59a9c4e392\" (UID: \"5359ba6b-7deb-4055-9069-1e59a9c4e392\") " Oct 30 22:19:47.700481 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:47.700423 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5359ba6b-7deb-4055-9069-1e59a9c4e392-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5359ba6b-7deb-4055-9069-1e59a9c4e392-istio-envoy\") pod \"5359ba6b-7deb-4055-9069-1e59a9c4e392\" (UID: \"5359ba6b-7deb-4055-9069-1e59a9c4e392\") " Oct 30 22:19:47.700481 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:47.700473 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5359ba6b-7deb-4055-9069-1e59a9c4e392-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5359ba6b-7deb-4055-9069-1e59a9c4e392-istio-podinfo\") pod \"5359ba6b-7deb-4055-9069-1e59a9c4e392\" (UID: \"5359ba6b-7deb-4055-9069-1e59a9c4e392\") " Oct 30 22:19:47.701239 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:47.701216 6720 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5359ba6b-7deb-4055-9069-1e59a9c4e392-workload-certs\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:47.701354 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:47.701244 6720 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5359ba6b-7deb-4055-9069-1e59a9c4e392-credential-socket\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:47.701569 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:47.701549 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5359ba6b-7deb-4055-9069-1e59a9c4e392-workload-socket" pod "5359ba6b-7deb-4055-9069-1e59a9c4e392" (UID: "5359ba6b-7deb-4055-9069-1e59a9c4e392"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:47.702097 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:47.702068 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5359ba6b-7deb-4055-9069-1e59a9c4e392-istiod-ca-cert" pod "5359ba6b-7deb-4055-9069-1e59a9c4e392" (UID: "5359ba6b-7deb-4055-9069-1e59a9c4e392"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:19:47.702492 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:47.702467 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5359ba6b-7deb-4055-9069-1e59a9c4e392-istio-data" pod "5359ba6b-7deb-4055-9069-1e59a9c4e392" (UID: "5359ba6b-7deb-4055-9069-1e59a9c4e392"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:47.702822 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:47.702793 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5359ba6b-7deb-4055-9069-1e59a9c4e392-istio-token" pod "5359ba6b-7deb-4055-9069-1e59a9c4e392" (UID: "5359ba6b-7deb-4055-9069-1e59a9c4e392"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:19:47.704491 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:47.704463 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5359ba6b-7deb-4055-9069-1e59a9c4e392-istio-envoy" pod "5359ba6b-7deb-4055-9069-1e59a9c4e392" (UID: "5359ba6b-7deb-4055-9069-1e59a9c4e392"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:47.705123 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:47.705091 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5359ba6b-7deb-4055-9069-1e59a9c4e392-istio-podinfo" pod "5359ba6b-7deb-4055-9069-1e59a9c4e392" (UID: "5359ba6b-7deb-4055-9069-1e59a9c4e392"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:19:47.706463 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:47.705769 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5359ba6b-7deb-4055-9069-1e59a9c4e392-kube-api-access-98jht" pod "5359ba6b-7deb-4055-9069-1e59a9c4e392" (UID: "5359ba6b-7deb-4055-9069-1e59a9c4e392"). InnerVolumeSpecName "kube-api-access-98jht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:19:47.802397 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:47.802344 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98jht\" (UniqueName: \"kubernetes.io/projected/5359ba6b-7deb-4055-9069-1e59a9c4e392-kube-api-access-98jht\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:47.802397 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:47.802399 6720 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5359ba6b-7deb-4055-9069-1e59a9c4e392-istio-data\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:47.802625 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:47.802414 6720 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5359ba6b-7deb-4055-9069-1e59a9c4e392-istiod-ca-cert\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:47.802625 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:47.802427 6720 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5359ba6b-7deb-4055-9069-1e59a9c4e392-istio-envoy\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:47.802625 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:47.802440 6720 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5359ba6b-7deb-4055-9069-1e59a9c4e392-istio-podinfo\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:47.802625 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:47.802452 6720 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5359ba6b-7deb-4055-9069-1e59a9c4e392-istio-token\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:47.802625 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:47.802464 6720 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5359ba6b-7deb-4055-9069-1e59a9c4e392-workload-socket\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:48.332703 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:48.332582 6720 generic.go:356] "Generic (PLEG): container finished" podID="5359ba6b-7deb-4055-9069-1e59a9c4e392" containerID="da85a37615c3f04fa1c21365d780a1fa28e97123dc6dea03c41dffa8a75c4963" exitCode=0 Oct 30 22:19:48.332703 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:48.332631 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-cjmnk" Oct 30 22:19:48.332703 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:48.332671 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-cjmnk" event={"ID":"5359ba6b-7deb-4055-9069-1e59a9c4e392","Type":"ContainerDied","Data":"da85a37615c3f04fa1c21365d780a1fa28e97123dc6dea03c41dffa8a75c4963"} Oct 30 22:19:48.333205 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:48.332750 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-cjmnk" event={"ID":"5359ba6b-7deb-4055-9069-1e59a9c4e392","Type":"ContainerDied","Data":"ab1da138b72e84fedaf338b2f560eb00b40365b94582a76b0077ddb33bd732d4"} Oct 30 22:19:48.333205 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:48.332785 6720 scope.go:122] "RemoveContainer" containerID="da85a37615c3f04fa1c21365d780a1fa28e97123dc6dea03c41dffa8a75c4963" Oct 30 22:19:48.343849 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:48.343833 6720 scope.go:122] "RemoveContainer" containerID="da85a37615c3f04fa1c21365d780a1fa28e97123dc6dea03c41dffa8a75c4963" Oct 30 22:19:48.344139 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:19:48.344120 6720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da85a37615c3f04fa1c21365d780a1fa28e97123dc6dea03c41dffa8a75c4963\": container with ID starting with da85a37615c3f04fa1c21365d780a1fa28e97123dc6dea03c41dffa8a75c4963 not found: ID does not exist" containerID="da85a37615c3f04fa1c21365d780a1fa28e97123dc6dea03c41dffa8a75c4963" Oct 30 22:19:48.344213 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:48.344152 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da85a37615c3f04fa1c21365d780a1fa28e97123dc6dea03c41dffa8a75c4963"} err="failed to get container status \"da85a37615c3f04fa1c21365d780a1fa28e97123dc6dea03c41dffa8a75c4963\": rpc error: code = NotFound desc = could not find container \"da85a37615c3f04fa1c21365d780a1fa28e97123dc6dea03c41dffa8a75c4963\": container with ID starting with da85a37615c3f04fa1c21365d780a1fa28e97123dc6dea03c41dffa8a75c4963 not found: ID does not exist" Oct 30 22:19:48.355446 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:48.355424 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-cjmnk"] Oct 30 22:19:48.359636 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:48.359612 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-cjmnk"] Oct 30 22:19:48.427365 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:48.427213 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:49.287377 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:49.287346 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:49.957934 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:49.957901 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5359ba6b-7deb-4055-9069-1e59a9c4e392" path="/var/lib/kubelet/pods/5359ba6b-7deb-4055-9069-1e59a9c4e392/volumes" Oct 30 22:19:50.091949 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:50.091915 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:50.744081 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:50.744052 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:50.906687 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:50.906653 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:51.638485 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:51.637573 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:52.457411 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:52.455913 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:53.208410 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:53.203918 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:53.848086 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:53.846511 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:54.635222 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:54.633085 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:55.384779 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:55.384001 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:56.075465 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:56.075425 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:56.740992 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:56.740953 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:57.431593 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:57.429693 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:57.623978 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:57.623944 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:57.780084 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:57.779989 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:57.920201 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:57.920174 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:58.098725 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:58.098638 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:58.497628 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:58.497572 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:58.677406 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:58.676147 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:58.909718 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:58.906624 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jhkhv"] Oct 30 22:19:58.909718 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:58.906920 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jhkhv" podUID="de3025de-693a-4527-a04e-c9dd6d3caf89" containerName="istio-proxy" containerID="cri-o://f4af5bcb39a183d6906be3f935d9f48a5e1fbd5fcbe4cb06dbd2d2d78f3be5dc" gracePeriod=30 Oct 30 22:19:59.109378 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:59.109310 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:59.273887 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:59.273747 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:19:59.918290 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:19:59.918230 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:00.083427 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:00.081830 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:00.498807 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:00.498778 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:01.297710 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:01.297671 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:02.151688 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:02.151655 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:03.033198 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:03.033152 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:03.759409 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:03.759358 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:04.220463 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:04.220438 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jhkhv" Oct 30 22:20:04.277586 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:04.277547 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de3025de-693a-4527-a04e-c9dd6d3caf89-kube-api-access-585ng\" (UniqueName: \"kubernetes.io/projected/de3025de-693a-4527-a04e-c9dd6d3caf89-kube-api-access-585ng\") pod \"de3025de-693a-4527-a04e-c9dd6d3caf89\" (UID: \"de3025de-693a-4527-a04e-c9dd6d3caf89\") " Oct 30 22:20:04.277790 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:04.277649 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de3025de-693a-4527-a04e-c9dd6d3caf89-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de3025de-693a-4527-a04e-c9dd6d3caf89-istio-envoy\") pod \"de3025de-693a-4527-a04e-c9dd6d3caf89\" (UID: \"de3025de-693a-4527-a04e-c9dd6d3caf89\") " Oct 30 22:20:04.277790 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:04.277728 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de3025de-693a-4527-a04e-c9dd6d3caf89-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de3025de-693a-4527-a04e-c9dd6d3caf89-istiod-ca-cert\") pod \"de3025de-693a-4527-a04e-c9dd6d3caf89\" (UID: \"de3025de-693a-4527-a04e-c9dd6d3caf89\") " Oct 30 22:20:04.277917 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:04.277809 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de3025de-693a-4527-a04e-c9dd6d3caf89-istio-token\" (UniqueName: \"kubernetes.io/projected/de3025de-693a-4527-a04e-c9dd6d3caf89-istio-token\") pod \"de3025de-693a-4527-a04e-c9dd6d3caf89\" (UID: \"de3025de-693a-4527-a04e-c9dd6d3caf89\") " Oct 30 22:20:04.277917 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:04.277867 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de3025de-693a-4527-a04e-c9dd6d3caf89-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de3025de-693a-4527-a04e-c9dd6d3caf89-workload-certs\") pod \"de3025de-693a-4527-a04e-c9dd6d3caf89\" (UID: \"de3025de-693a-4527-a04e-c9dd6d3caf89\") " Oct 30 22:20:04.278038 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:04.277958 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de3025de-693a-4527-a04e-c9dd6d3caf89-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de3025de-693a-4527-a04e-c9dd6d3caf89-credential-socket\") pod \"de3025de-693a-4527-a04e-c9dd6d3caf89\" (UID: \"de3025de-693a-4527-a04e-c9dd6d3caf89\") " Oct 30 22:20:04.278038 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:04.277991 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/de3025de-693a-4527-a04e-c9dd6d3caf89-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de3025de-693a-4527-a04e-c9dd6d3caf89-istio-podinfo\") pod \"de3025de-693a-4527-a04e-c9dd6d3caf89\" (UID: \"de3025de-693a-4527-a04e-c9dd6d3caf89\") " Oct 30 22:20:04.278140 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:04.278082 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de3025de-693a-4527-a04e-c9dd6d3caf89-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de3025de-693a-4527-a04e-c9dd6d3caf89-workload-socket\") pod \"de3025de-693a-4527-a04e-c9dd6d3caf89\" (UID: \"de3025de-693a-4527-a04e-c9dd6d3caf89\") " Oct 30 22:20:04.278208 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:04.278189 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de3025de-693a-4527-a04e-c9dd6d3caf89-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de3025de-693a-4527-a04e-c9dd6d3caf89-istio-data\") pod \"de3025de-693a-4527-a04e-c9dd6d3caf89\" (UID: \"de3025de-693a-4527-a04e-c9dd6d3caf89\") " Oct 30 22:20:04.278578 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:04.278311 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de3025de-693a-4527-a04e-c9dd6d3caf89-istiod-ca-cert" pod "de3025de-693a-4527-a04e-c9dd6d3caf89" (UID: "de3025de-693a-4527-a04e-c9dd6d3caf89"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:20:04.278949 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:04.278756 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de3025de-693a-4527-a04e-c9dd6d3caf89-credential-socket" pod "de3025de-693a-4527-a04e-c9dd6d3caf89" (UID: "de3025de-693a-4527-a04e-c9dd6d3caf89"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:20:04.279041 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:04.279000 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de3025de-693a-4527-a04e-c9dd6d3caf89-istio-data" pod "de3025de-693a-4527-a04e-c9dd6d3caf89" (UID: "de3025de-693a-4527-a04e-c9dd6d3caf89"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:20:04.279411 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:04.279180 6720 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de3025de-693a-4527-a04e-c9dd6d3caf89-istiod-ca-cert\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:20:04.279411 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:04.279210 6720 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de3025de-693a-4527-a04e-c9dd6d3caf89-credential-socket\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:20:04.279956 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:04.279925 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de3025de-693a-4527-a04e-c9dd6d3caf89-workload-socket" pod "de3025de-693a-4527-a04e-c9dd6d3caf89" (UID: "de3025de-693a-4527-a04e-c9dd6d3caf89"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:20:04.279956 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:04.279942 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de3025de-693a-4527-a04e-c9dd6d3caf89-workload-certs" pod "de3025de-693a-4527-a04e-c9dd6d3caf89" (UID: "de3025de-693a-4527-a04e-c9dd6d3caf89"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:20:04.281514 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:04.281486 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de3025de-693a-4527-a04e-c9dd6d3caf89-kube-api-access-585ng" pod "de3025de-693a-4527-a04e-c9dd6d3caf89" (UID: "de3025de-693a-4527-a04e-c9dd6d3caf89"). InnerVolumeSpecName "kube-api-access-585ng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:20:04.282314 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:04.282288 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de3025de-693a-4527-a04e-c9dd6d3caf89-istio-envoy" pod "de3025de-693a-4527-a04e-c9dd6d3caf89" (UID: "de3025de-693a-4527-a04e-c9dd6d3caf89"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:20:04.282741 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:04.282707 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/de3025de-693a-4527-a04e-c9dd6d3caf89-istio-podinfo" pod "de3025de-693a-4527-a04e-c9dd6d3caf89" (UID: "de3025de-693a-4527-a04e-c9dd6d3caf89"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:20:04.283882 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:04.283857 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de3025de-693a-4527-a04e-c9dd6d3caf89-istio-token" pod "de3025de-693a-4527-a04e-c9dd6d3caf89" (UID: "de3025de-693a-4527-a04e-c9dd6d3caf89"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:20:04.380363 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:04.380331 6720 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de3025de-693a-4527-a04e-c9dd6d3caf89-workload-certs\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:20:04.380363 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:04.380362 6720 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de3025de-693a-4527-a04e-c9dd6d3caf89-istio-podinfo\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:20:04.380363 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:04.380371 6720 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de3025de-693a-4527-a04e-c9dd6d3caf89-workload-socket\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:20:04.380670 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:04.380380 6720 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de3025de-693a-4527-a04e-c9dd6d3caf89-istio-data\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:20:04.380670 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:04.380424 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-585ng\" (UniqueName: \"kubernetes.io/projected/de3025de-693a-4527-a04e-c9dd6d3caf89-kube-api-access-585ng\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:20:04.380670 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:04.380434 6720 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de3025de-693a-4527-a04e-c9dd6d3caf89-istio-envoy\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:20:04.380670 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:04.380443 6720 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de3025de-693a-4527-a04e-c9dd6d3caf89-istio-token\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:20:04.517227 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:04.516044 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:04.550181 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:04.550145 6720 generic.go:356] "Generic (PLEG): container finished" podID="de3025de-693a-4527-a04e-c9dd6d3caf89" containerID="f4af5bcb39a183d6906be3f935d9f48a5e1fbd5fcbe4cb06dbd2d2d78f3be5dc" exitCode=0 Oct 30 22:20:04.550444 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:04.550199 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jhkhv" Oct 30 22:20:04.550444 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:04.550224 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jhkhv" event={"ID":"de3025de-693a-4527-a04e-c9dd6d3caf89","Type":"ContainerDied","Data":"f4af5bcb39a183d6906be3f935d9f48a5e1fbd5fcbe4cb06dbd2d2d78f3be5dc"} Oct 30 22:20:04.550444 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:04.550284 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jhkhv" event={"ID":"de3025de-693a-4527-a04e-c9dd6d3caf89","Type":"ContainerDied","Data":"c6b81f151d530d07d39e98300ebe602916af6ebf71f968b96ce260a7f29140ba"} Oct 30 22:20:04.550615 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:04.550367 6720 scope.go:122] "RemoveContainer" containerID="f4af5bcb39a183d6906be3f935d9f48a5e1fbd5fcbe4cb06dbd2d2d78f3be5dc" Oct 30 22:20:04.570556 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:04.569940 6720 scope.go:122] "RemoveContainer" containerID="f4af5bcb39a183d6906be3f935d9f48a5e1fbd5fcbe4cb06dbd2d2d78f3be5dc" Oct 30 22:20:04.570928 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:20:04.570900 6720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4af5bcb39a183d6906be3f935d9f48a5e1fbd5fcbe4cb06dbd2d2d78f3be5dc\": container with ID starting with f4af5bcb39a183d6906be3f935d9f48a5e1fbd5fcbe4cb06dbd2d2d78f3be5dc not found: ID does not exist" containerID="f4af5bcb39a183d6906be3f935d9f48a5e1fbd5fcbe4cb06dbd2d2d78f3be5dc" Oct 30 22:20:04.571044 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:04.570944 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4af5bcb39a183d6906be3f935d9f48a5e1fbd5fcbe4cb06dbd2d2d78f3be5dc"} err="failed to get container status \"f4af5bcb39a183d6906be3f935d9f48a5e1fbd5fcbe4cb06dbd2d2d78f3be5dc\": rpc error: code = NotFound desc = could not find container \"f4af5bcb39a183d6906be3f935d9f48a5e1fbd5fcbe4cb06dbd2d2d78f3be5dc\": container with ID starting with f4af5bcb39a183d6906be3f935d9f48a5e1fbd5fcbe4cb06dbd2d2d78f3be5dc not found: ID does not exist" Oct 30 22:20:04.581797 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:04.581762 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jhkhv"] Oct 30 22:20:04.587348 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:04.587319 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jhkhv"] Oct 30 22:20:05.266044 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:05.265989 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:05.951293 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:05.947953 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:05.975446 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:05.975008 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de3025de-693a-4527-a04e-c9dd6d3caf89" path="/var/lib/kubelet/pods/de3025de-693a-4527-a04e-c9dd6d3caf89/volumes" Oct 30 22:20:06.654328 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:06.654283 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:07.401424 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:07.401373 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:08.045536 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:08.045494 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:08.669975 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:08.669940 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:09.379987 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:09.379949 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:09.936021 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:09.935950 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:10.404731 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:20:10.404696 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 22:20:10.524339 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:10.524300 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl"] Oct 30 22:20:10.524573 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:10.524546 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl" podUID="63d87167-397f-476b-837f-e40c8f14a994" containerName="istio-proxy" containerID="cri-o://f17f342c14597f7e83fac1551383c507e2739e0a757e7c889d2cd49a1f890edd" gracePeriod=30 Oct 30 22:20:10.755809 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:10.755685 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:11.461409 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:11.458469 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:12.131409 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:12.121705 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:12.767956 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:12.767902 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:13.461419 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:13.461325 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:14.262804 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:14.261976 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:15.015539 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:15.011929 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:15.047472 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:15.039076 6720 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:20:15.047472 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:15.039135 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl" podUID="63d87167-397f-476b-837f-e40c8f14a994" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:20:15.796706 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:15.796614 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:15.859605 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:15.859576 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl" Oct 30 22:20:15.938367 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:15.938335 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63d87167-397f-476b-837f-e40c8f14a994-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63d87167-397f-476b-837f-e40c8f14a994-workload-certs\") pod \"63d87167-397f-476b-837f-e40c8f14a994\" (UID: \"63d87167-397f-476b-837f-e40c8f14a994\") " Oct 30 22:20:15.938576 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:15.938398 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63d87167-397f-476b-837f-e40c8f14a994-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63d87167-397f-476b-837f-e40c8f14a994-credential-socket\") pod \"63d87167-397f-476b-837f-e40c8f14a994\" (UID: \"63d87167-397f-476b-837f-e40c8f14a994\") " Oct 30 22:20:15.938576 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:15.938442 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63d87167-397f-476b-837f-e40c8f14a994-kube-api-access-tcfxt\" (UniqueName: \"kubernetes.io/projected/63d87167-397f-476b-837f-e40c8f14a994-kube-api-access-tcfxt\") pod \"63d87167-397f-476b-837f-e40c8f14a994\" (UID: \"63d87167-397f-476b-837f-e40c8f14a994\") " Oct 30 22:20:15.938576 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:15.938471 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63d87167-397f-476b-837f-e40c8f14a994-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63d87167-397f-476b-837f-e40c8f14a994-istio-data\") pod \"63d87167-397f-476b-837f-e40c8f14a994\" (UID: \"63d87167-397f-476b-837f-e40c8f14a994\") " Oct 30 22:20:15.938821 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:15.938587 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63d87167-397f-476b-837f-e40c8f14a994-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63d87167-397f-476b-837f-e40c8f14a994-workload-socket\") pod \"63d87167-397f-476b-837f-e40c8f14a994\" (UID: \"63d87167-397f-476b-837f-e40c8f14a994\") " Oct 30 22:20:15.938821 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:15.938660 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63d87167-397f-476b-837f-e40c8f14a994-workload-certs" pod "63d87167-397f-476b-837f-e40c8f14a994" (UID: "63d87167-397f-476b-837f-e40c8f14a994"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:20:15.938821 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:15.938676 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63d87167-397f-476b-837f-e40c8f14a994-istio-token\" (UniqueName: \"kubernetes.io/projected/63d87167-397f-476b-837f-e40c8f14a994-istio-token\") pod \"63d87167-397f-476b-837f-e40c8f14a994\" (UID: \"63d87167-397f-476b-837f-e40c8f14a994\") " Oct 30 22:20:15.938821 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:15.938751 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63d87167-397f-476b-837f-e40c8f14a994-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63d87167-397f-476b-837f-e40c8f14a994-istio-envoy\") pod \"63d87167-397f-476b-837f-e40c8f14a994\" (UID: \"63d87167-397f-476b-837f-e40c8f14a994\") " Oct 30 22:20:15.938821 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:15.938792 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63d87167-397f-476b-837f-e40c8f14a994-credential-socket" pod "63d87167-397f-476b-837f-e40c8f14a994" (UID: "63d87167-397f-476b-837f-e40c8f14a994"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:20:15.939093 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:15.938804 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63d87167-397f-476b-837f-e40c8f14a994-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63d87167-397f-476b-837f-e40c8f14a994-istiod-ca-cert\") pod \"63d87167-397f-476b-837f-e40c8f14a994\" (UID: \"63d87167-397f-476b-837f-e40c8f14a994\") " Oct 30 22:20:15.939093 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:15.938868 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63d87167-397f-476b-837f-e40c8f14a994-istio-data" pod "63d87167-397f-476b-837f-e40c8f14a994" (UID: "63d87167-397f-476b-837f-e40c8f14a994"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:20:15.939093 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:15.938949 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/63d87167-397f-476b-837f-e40c8f14a994-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63d87167-397f-476b-837f-e40c8f14a994-istio-podinfo\") pod \"63d87167-397f-476b-837f-e40c8f14a994\" (UID: \"63d87167-397f-476b-837f-e40c8f14a994\") " Oct 30 22:20:15.939399 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:15.939133 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63d87167-397f-476b-837f-e40c8f14a994-workload-socket" pod "63d87167-397f-476b-837f-e40c8f14a994" (UID: "63d87167-397f-476b-837f-e40c8f14a994"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:20:15.939399 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:15.939148 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63d87167-397f-476b-837f-e40c8f14a994-istiod-ca-cert" pod "63d87167-397f-476b-837f-e40c8f14a994" (UID: "63d87167-397f-476b-837f-e40c8f14a994"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:20:15.939691 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:15.939667 6720 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63d87167-397f-476b-837f-e40c8f14a994-credential-socket\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:20:15.939873 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:15.939849 6720 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63d87167-397f-476b-837f-e40c8f14a994-istio-data\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:20:15.939973 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:15.939877 6720 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63d87167-397f-476b-837f-e40c8f14a994-workload-socket\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:20:15.939973 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:15.939892 6720 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63d87167-397f-476b-837f-e40c8f14a994-istiod-ca-cert\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:20:15.939973 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:15.939905 6720 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63d87167-397f-476b-837f-e40c8f14a994-workload-certs\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:20:15.941505 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:15.941463 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63d87167-397f-476b-837f-e40c8f14a994-kube-api-access-tcfxt" pod "63d87167-397f-476b-837f-e40c8f14a994" (UID: "63d87167-397f-476b-837f-e40c8f14a994"). InnerVolumeSpecName "kube-api-access-tcfxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:20:15.942344 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:15.942297 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/63d87167-397f-476b-837f-e40c8f14a994-istio-podinfo" pod "63d87167-397f-476b-837f-e40c8f14a994" (UID: "63d87167-397f-476b-837f-e40c8f14a994"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:20:15.942902 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:15.942860 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63d87167-397f-476b-837f-e40c8f14a994-istio-envoy" pod "63d87167-397f-476b-837f-e40c8f14a994" (UID: "63d87167-397f-476b-837f-e40c8f14a994"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:20:15.943018 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:15.942949 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63d87167-397f-476b-837f-e40c8f14a994-istio-token" pod "63d87167-397f-476b-837f-e40c8f14a994" (UID: "63d87167-397f-476b-837f-e40c8f14a994"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:20:16.041669 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:16.041629 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcfxt\" (UniqueName: \"kubernetes.io/projected/63d87167-397f-476b-837f-e40c8f14a994-kube-api-access-tcfxt\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:20:16.041669 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:16.041658 6720 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63d87167-397f-476b-837f-e40c8f14a994-istio-token\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:20:16.041669 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:16.041668 6720 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63d87167-397f-476b-837f-e40c8f14a994-istio-envoy\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:20:16.041669 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:16.041679 6720 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63d87167-397f-476b-837f-e40c8f14a994-istio-podinfo\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:20:16.635595 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:16.635548 6720 generic.go:356] "Generic (PLEG): container finished" podID="63d87167-397f-476b-837f-e40c8f14a994" containerID="f17f342c14597f7e83fac1551383c507e2739e0a757e7c889d2cd49a1f890edd" exitCode=0 Oct 30 22:20:16.635807 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:16.635629 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl" Oct 30 22:20:16.635807 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:16.635632 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl" event={"ID":"63d87167-397f-476b-837f-e40c8f14a994","Type":"ContainerDied","Data":"f17f342c14597f7e83fac1551383c507e2739e0a757e7c889d2cd49a1f890edd"} Oct 30 22:20:16.635807 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:16.635781 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl" event={"ID":"63d87167-397f-476b-837f-e40c8f14a994","Type":"ContainerDied","Data":"d9ca1487294c6ece545148dfb5e75c599d7425da7d45e443fa0a0f742462d656"} Oct 30 22:20:16.635807 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:16.635806 6720 scope.go:122] "RemoveContainer" containerID="f17f342c14597f7e83fac1551383c507e2739e0a757e7c889d2cd49a1f890edd" Oct 30 22:20:16.649234 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:16.649060 6720 scope.go:122] "RemoveContainer" containerID="f17f342c14597f7e83fac1551383c507e2739e0a757e7c889d2cd49a1f890edd" Oct 30 22:20:16.649475 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:20:16.649444 6720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f17f342c14597f7e83fac1551383c507e2739e0a757e7c889d2cd49a1f890edd\": container with ID starting with f17f342c14597f7e83fac1551383c507e2739e0a757e7c889d2cd49a1f890edd not found: ID does not exist" containerID="f17f342c14597f7e83fac1551383c507e2739e0a757e7c889d2cd49a1f890edd" Oct 30 22:20:16.649587 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:16.649488 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f17f342c14597f7e83fac1551383c507e2739e0a757e7c889d2cd49a1f890edd"} err="failed to get container status \"f17f342c14597f7e83fac1551383c507e2739e0a757e7c889d2cd49a1f890edd\": rpc error: code = NotFound desc = could not find container \"f17f342c14597f7e83fac1551383c507e2739e0a757e7c889d2cd49a1f890edd\": container with ID starting with f17f342c14597f7e83fac1551383c507e2739e0a757e7c889d2cd49a1f890edd not found: ID does not exist" Oct 30 22:20:16.675799 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:16.675765 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl"] Oct 30 22:20:16.680428 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:16.680369 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-4ntgl"] Oct 30 22:20:16.841414 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:16.841367 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:17.957612 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:17.957582 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63d87167-397f-476b-837f-e40c8f14a994" path="/var/lib/kubelet/pods/63d87167-397f-476b-837f-e40c8f14a994/volumes" Oct 30 22:20:18.033015 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:18.032979 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:18.758327 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:18.757921 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:19.424420 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:19.416699 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:20.154979 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:20.154939 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:20.906494 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:20.906455 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:21.655941 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:21.655895 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:22.430514 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:22.430470 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:23.108414 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:23.105271 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:23.794136 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:23.794103 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:24.441343 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:24.441305 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:24.919723 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:24.919690 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:25.135556 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:25.135443 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:25.744742 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:25.744681 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:26.448464 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:26.448424 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:27.141227 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:27.141190 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:27.823301 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:27.823268 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:27.978991 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:27.978961 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:28.569691 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:28.569655 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:29.140906 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:29.140868 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:29.376466 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:29.376435 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:29.843528 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:29.843482 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:30.558707 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:30.558675 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:31.260008 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:31.259969 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:32.126344 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:32.126309 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:32.858851 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:32.858800 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:33.641306 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:33.641271 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:34.355911 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:34.355425 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:35.033490 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:35.033332 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:35.720409 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:35.719038 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:36.355567 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:36.355462 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:37.058225 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:37.058166 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:37.735321 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:37.735282 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:38.352278 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:38.352241 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:39.031406 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:39.027931 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:39.737556 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:39.723055 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:40.403867 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:20:40.403830 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 22:20:40.494137 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:40.494089 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:41.168856 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:41.168792 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:41.887016 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:41.886968 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:42.612917 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:42.612882 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:43.317884 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:43.317847 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:43.971022 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:43.969220 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:44.670377 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:44.670347 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:45.404412 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:45.400154 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:46.129345 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:46.129296 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:46.780676 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:46.780254 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:47.572485 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:47.572377 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:48.426026 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:48.425985 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:49.031482 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:49.031451 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:49.741792 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:49.741695 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:50.605674 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:50.605628 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:51.185823 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:51.185782 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:51.937228 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:51.937169 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:52.685418 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:52.685327 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:53.345446 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:53.328910 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:53.992006 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:53.991953 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:54.654225 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:54.654187 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:55.302407 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:55.301195 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:56.077672 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:56.077585 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:56.761416 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:56.760986 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:57.407283 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:57.407241 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:58.035136 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:58.035099 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:58.742953 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:58.742904 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:20:59.384712 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:20:59.384675 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:00.049029 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:00.048992 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:00.742240 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:00.742206 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:01.350144 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:01.347198 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:02.211349 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:02.211293 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:02.829419 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:02.827899 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:03.465112 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:03.462807 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:04.106689 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:04.106655 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:04.427046 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:04.426971 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:04.590508 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:04.590423 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:05.204409 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:05.203128 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:05.421197 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:05.421165 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-n5hzj"] Oct 30 22:21:05.421483 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:05.421452 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-n5hzj" podUID="28b34a7e-4765-46d0-b412-74eb9a1567d9" containerName="istio-proxy" containerID="cri-o://2c26c70d220b333ea4dbba0c8cc1cbe4828487c2e752f86fa8d5090f6b391bf1" gracePeriod=30 Oct 30 22:21:05.425297 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:05.425276 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-wc924"] Oct 30 22:21:05.425524 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:05.425499 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-wc924" podUID="0051f207-fce7-4aed-bcc3-a5aa69bb9a3c" containerName="istio-proxy" containerID="cri-o://76650545cd685ef5bfdeebaaa3a26dbf09d04f9afff648f4fa051bd6185ecf4e" gracePeriod=30 Oct 30 22:21:05.920432 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:05.919990 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:06.364177 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:06.364144 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:06.505951 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:06.505910 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:06.648713 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:06.648635 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:07.381707 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:07.381656 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:08.023572 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:08.023516 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:08.667734 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:08.667682 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:09.342406 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:09.342300 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:10.015048 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:10.015012 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:10.405132 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:21:10.405094 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 22:21:10.663776 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:10.655486 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:10.771681 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:10.771658 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-n5hzj" Oct 30 22:21:10.836813 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:10.836781 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28b34a7e-4765-46d0-b412-74eb9a1567d9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28b34a7e-4765-46d0-b412-74eb9a1567d9-istio-envoy\") pod \"28b34a7e-4765-46d0-b412-74eb9a1567d9\" (UID: \"28b34a7e-4765-46d0-b412-74eb9a1567d9\") " Oct 30 22:21:10.837001 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:10.836868 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28b34a7e-4765-46d0-b412-74eb9a1567d9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28b34a7e-4765-46d0-b412-74eb9a1567d9-istiod-ca-cert\") pod \"28b34a7e-4765-46d0-b412-74eb9a1567d9\" (UID: \"28b34a7e-4765-46d0-b412-74eb9a1567d9\") " Oct 30 22:21:10.837079 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:10.837030 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28b34a7e-4765-46d0-b412-74eb9a1567d9-istio-token\" (UniqueName: \"kubernetes.io/projected/28b34a7e-4765-46d0-b412-74eb9a1567d9-istio-token\") pod \"28b34a7e-4765-46d0-b412-74eb9a1567d9\" (UID: \"28b34a7e-4765-46d0-b412-74eb9a1567d9\") " Oct 30 22:21:10.837171 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:10.837151 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28b34a7e-4765-46d0-b412-74eb9a1567d9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28b34a7e-4765-46d0-b412-74eb9a1567d9-credential-socket\") pod \"28b34a7e-4765-46d0-b412-74eb9a1567d9\" (UID: \"28b34a7e-4765-46d0-b412-74eb9a1567d9\") " Oct 30 22:21:10.837251 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:10.837205 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28b34a7e-4765-46d0-b412-74eb9a1567d9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28b34a7e-4765-46d0-b412-74eb9a1567d9-istio-data\") pod \"28b34a7e-4765-46d0-b412-74eb9a1567d9\" (UID: \"28b34a7e-4765-46d0-b412-74eb9a1567d9\") " Oct 30 22:21:10.837251 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:10.837226 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28b34a7e-4765-46d0-b412-74eb9a1567d9-kube-api-access-k4k7v\" (UniqueName: \"kubernetes.io/projected/28b34a7e-4765-46d0-b412-74eb9a1567d9-kube-api-access-k4k7v\") pod \"28b34a7e-4765-46d0-b412-74eb9a1567d9\" (UID: \"28b34a7e-4765-46d0-b412-74eb9a1567d9\") " Oct 30 22:21:10.837347 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:10.837262 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/28b34a7e-4765-46d0-b412-74eb9a1567d9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28b34a7e-4765-46d0-b412-74eb9a1567d9-istio-podinfo\") pod \"28b34a7e-4765-46d0-b412-74eb9a1567d9\" (UID: \"28b34a7e-4765-46d0-b412-74eb9a1567d9\") " Oct 30 22:21:10.837454 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:10.837434 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28b34a7e-4765-46d0-b412-74eb9a1567d9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28b34a7e-4765-46d0-b412-74eb9a1567d9-workload-certs\") pod \"28b34a7e-4765-46d0-b412-74eb9a1567d9\" (UID: \"28b34a7e-4765-46d0-b412-74eb9a1567d9\") " Oct 30 22:21:10.837523 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:10.837472 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28b34a7e-4765-46d0-b412-74eb9a1567d9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28b34a7e-4765-46d0-b412-74eb9a1567d9-workload-socket\") pod \"28b34a7e-4765-46d0-b412-74eb9a1567d9\" (UID: \"28b34a7e-4765-46d0-b412-74eb9a1567d9\") " Oct 30 22:21:10.837749 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:10.837517 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28b34a7e-4765-46d0-b412-74eb9a1567d9-istiod-ca-cert" pod "28b34a7e-4765-46d0-b412-74eb9a1567d9" (UID: "28b34a7e-4765-46d0-b412-74eb9a1567d9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:21:10.838241 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:10.838220 6720 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28b34a7e-4765-46d0-b412-74eb9a1567d9-istiod-ca-cert\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:10.838492 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:10.838469 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28b34a7e-4765-46d0-b412-74eb9a1567d9-workload-socket" pod "28b34a7e-4765-46d0-b412-74eb9a1567d9" (UID: "28b34a7e-4765-46d0-b412-74eb9a1567d9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:21:10.839365 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:10.839332 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28b34a7e-4765-46d0-b412-74eb9a1567d9-istio-envoy" pod "28b34a7e-4765-46d0-b412-74eb9a1567d9" (UID: "28b34a7e-4765-46d0-b412-74eb9a1567d9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:21:10.840455 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:10.840369 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-wc924" Oct 30 22:21:10.840828 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:10.840808 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28b34a7e-4765-46d0-b412-74eb9a1567d9-credential-socket" pod "28b34a7e-4765-46d0-b412-74eb9a1567d9" (UID: "28b34a7e-4765-46d0-b412-74eb9a1567d9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:21:10.841049 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:10.841024 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28b34a7e-4765-46d0-b412-74eb9a1567d9-istio-data" pod "28b34a7e-4765-46d0-b412-74eb9a1567d9" (UID: "28b34a7e-4765-46d0-b412-74eb9a1567d9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:21:10.841152 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:10.841051 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28b34a7e-4765-46d0-b412-74eb9a1567d9-workload-certs" pod "28b34a7e-4765-46d0-b412-74eb9a1567d9" (UID: "28b34a7e-4765-46d0-b412-74eb9a1567d9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:21:10.841152 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:10.841069 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28b34a7e-4765-46d0-b412-74eb9a1567d9-istio-token" pod "28b34a7e-4765-46d0-b412-74eb9a1567d9" (UID: "28b34a7e-4765-46d0-b412-74eb9a1567d9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:21:10.841905 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:10.841879 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28b34a7e-4765-46d0-b412-74eb9a1567d9-kube-api-access-k4k7v" pod "28b34a7e-4765-46d0-b412-74eb9a1567d9" (UID: "28b34a7e-4765-46d0-b412-74eb9a1567d9"). InnerVolumeSpecName "kube-api-access-k4k7v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:21:10.843356 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:10.843319 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/28b34a7e-4765-46d0-b412-74eb9a1567d9-istio-podinfo" pod "28b34a7e-4765-46d0-b412-74eb9a1567d9" (UID: "28b34a7e-4765-46d0-b412-74eb9a1567d9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:21:10.939225 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:10.939128 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0051f207-fce7-4aed-bcc3-a5aa69bb9a3c-istio-token\" (UniqueName: \"kubernetes.io/projected/0051f207-fce7-4aed-bcc3-a5aa69bb9a3c-istio-token\") pod \"0051f207-fce7-4aed-bcc3-a5aa69bb9a3c\" (UID: \"0051f207-fce7-4aed-bcc3-a5aa69bb9a3c\") " Oct 30 22:21:10.939225 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:10.939173 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0051f207-fce7-4aed-bcc3-a5aa69bb9a3c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0051f207-fce7-4aed-bcc3-a5aa69bb9a3c-istio-podinfo\") pod \"0051f207-fce7-4aed-bcc3-a5aa69bb9a3c\" (UID: \"0051f207-fce7-4aed-bcc3-a5aa69bb9a3c\") " Oct 30 22:21:10.939641 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:10.939279 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0051f207-fce7-4aed-bcc3-a5aa69bb9a3c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0051f207-fce7-4aed-bcc3-a5aa69bb9a3c-istio-data\") pod \"0051f207-fce7-4aed-bcc3-a5aa69bb9a3c\" (UID: \"0051f207-fce7-4aed-bcc3-a5aa69bb9a3c\") " Oct 30 22:21:10.939641 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:10.939609 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0051f207-fce7-4aed-bcc3-a5aa69bb9a3c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0051f207-fce7-4aed-bcc3-a5aa69bb9a3c-credential-socket\") pod \"0051f207-fce7-4aed-bcc3-a5aa69bb9a3c\" (UID: \"0051f207-fce7-4aed-bcc3-a5aa69bb9a3c\") " Oct 30 22:21:10.939760 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:10.939652 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0051f207-fce7-4aed-bcc3-a5aa69bb9a3c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0051f207-fce7-4aed-bcc3-a5aa69bb9a3c-workload-socket\") pod \"0051f207-fce7-4aed-bcc3-a5aa69bb9a3c\" (UID: \"0051f207-fce7-4aed-bcc3-a5aa69bb9a3c\") " Oct 30 22:21:10.939760 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:10.939670 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0051f207-fce7-4aed-bcc3-a5aa69bb9a3c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0051f207-fce7-4aed-bcc3-a5aa69bb9a3c-istiod-ca-cert\") pod \"0051f207-fce7-4aed-bcc3-a5aa69bb9a3c\" (UID: \"0051f207-fce7-4aed-bcc3-a5aa69bb9a3c\") " Oct 30 22:21:10.939870 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:10.939770 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0051f207-fce7-4aed-bcc3-a5aa69bb9a3c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0051f207-fce7-4aed-bcc3-a5aa69bb9a3c-istio-envoy\") pod \"0051f207-fce7-4aed-bcc3-a5aa69bb9a3c\" (UID: \"0051f207-fce7-4aed-bcc3-a5aa69bb9a3c\") " Oct 30 22:21:10.939870 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:10.939829 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0051f207-fce7-4aed-bcc3-a5aa69bb9a3c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0051f207-fce7-4aed-bcc3-a5aa69bb9a3c-workload-certs\") pod \"0051f207-fce7-4aed-bcc3-a5aa69bb9a3c\" (UID: \"0051f207-fce7-4aed-bcc3-a5aa69bb9a3c\") " Oct 30 22:21:10.940143 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:10.939912 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0051f207-fce7-4aed-bcc3-a5aa69bb9a3c-kube-api-access-8svxh\" (UniqueName: \"kubernetes.io/projected/0051f207-fce7-4aed-bcc3-a5aa69bb9a3c-kube-api-access-8svxh\") pod \"0051f207-fce7-4aed-bcc3-a5aa69bb9a3c\" (UID: \"0051f207-fce7-4aed-bcc3-a5aa69bb9a3c\") " Oct 30 22:21:10.940143 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:10.939923 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0051f207-fce7-4aed-bcc3-a5aa69bb9a3c-credential-socket" pod "0051f207-fce7-4aed-bcc3-a5aa69bb9a3c" (UID: "0051f207-fce7-4aed-bcc3-a5aa69bb9a3c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:21:10.940143 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:10.939979 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0051f207-fce7-4aed-bcc3-a5aa69bb9a3c-workload-socket" pod "0051f207-fce7-4aed-bcc3-a5aa69bb9a3c" (UID: "0051f207-fce7-4aed-bcc3-a5aa69bb9a3c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:21:10.940143 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:10.940042 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0051f207-fce7-4aed-bcc3-a5aa69bb9a3c-istio-data" pod "0051f207-fce7-4aed-bcc3-a5aa69bb9a3c" (UID: "0051f207-fce7-4aed-bcc3-a5aa69bb9a3c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:21:10.940365 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:10.940261 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0051f207-fce7-4aed-bcc3-a5aa69bb9a3c-workload-certs" pod "0051f207-fce7-4aed-bcc3-a5aa69bb9a3c" (UID: "0051f207-fce7-4aed-bcc3-a5aa69bb9a3c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:21:10.940638 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:10.940610 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0051f207-fce7-4aed-bcc3-a5aa69bb9a3c-istiod-ca-cert" pod "0051f207-fce7-4aed-bcc3-a5aa69bb9a3c" (UID: "0051f207-fce7-4aed-bcc3-a5aa69bb9a3c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:21:10.940711 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:10.940666 6720 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28b34a7e-4765-46d0-b412-74eb9a1567d9-istio-podinfo\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:10.940711 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:10.940689 6720 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28b34a7e-4765-46d0-b412-74eb9a1567d9-workload-certs\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:10.940711 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:10.940705 6720 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28b34a7e-4765-46d0-b412-74eb9a1567d9-workload-socket\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:10.940874 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:10.940718 6720 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28b34a7e-4765-46d0-b412-74eb9a1567d9-istio-envoy\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:10.940874 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:10.940733 6720 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0051f207-fce7-4aed-bcc3-a5aa69bb9a3c-istio-data\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:10.940874 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:10.940747 6720 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0051f207-fce7-4aed-bcc3-a5aa69bb9a3c-credential-socket\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:10.940874 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:10.940760 6720 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0051f207-fce7-4aed-bcc3-a5aa69bb9a3c-workload-socket\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:10.940874 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:10.940774 6720 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28b34a7e-4765-46d0-b412-74eb9a1567d9-istio-token\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:10.940874 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:10.940785 6720 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0051f207-fce7-4aed-bcc3-a5aa69bb9a3c-workload-certs\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:10.940874 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:10.940797 6720 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28b34a7e-4765-46d0-b412-74eb9a1567d9-credential-socket\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:10.940874 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:10.940811 6720 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28b34a7e-4765-46d0-b412-74eb9a1567d9-istio-data\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:10.940874 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:10.940825 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4k7v\" (UniqueName: \"kubernetes.io/projected/28b34a7e-4765-46d0-b412-74eb9a1567d9-kube-api-access-k4k7v\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:10.941977 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:10.941954 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0051f207-fce7-4aed-bcc3-a5aa69bb9a3c-istio-token" pod "0051f207-fce7-4aed-bcc3-a5aa69bb9a3c" (UID: "0051f207-fce7-4aed-bcc3-a5aa69bb9a3c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:21:10.942832 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:10.942811 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0051f207-fce7-4aed-bcc3-a5aa69bb9a3c-kube-api-access-8svxh" pod "0051f207-fce7-4aed-bcc3-a5aa69bb9a3c" (UID: "0051f207-fce7-4aed-bcc3-a5aa69bb9a3c"). InnerVolumeSpecName "kube-api-access-8svxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:21:10.942832 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:10.942814 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0051f207-fce7-4aed-bcc3-a5aa69bb9a3c-istio-envoy" pod "0051f207-fce7-4aed-bcc3-a5aa69bb9a3c" (UID: "0051f207-fce7-4aed-bcc3-a5aa69bb9a3c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:21:10.943373 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:10.943357 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0051f207-fce7-4aed-bcc3-a5aa69bb9a3c-istio-podinfo" pod "0051f207-fce7-4aed-bcc3-a5aa69bb9a3c" (UID: "0051f207-fce7-4aed-bcc3-a5aa69bb9a3c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:21:11.042622 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:11.042588 6720 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0051f207-fce7-4aed-bcc3-a5aa69bb9a3c-istio-token\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:11.042622 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:11.042615 6720 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0051f207-fce7-4aed-bcc3-a5aa69bb9a3c-istio-podinfo\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:11.042622 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:11.042625 6720 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0051f207-fce7-4aed-bcc3-a5aa69bb9a3c-istiod-ca-cert\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:11.042622 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:11.042633 6720 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0051f207-fce7-4aed-bcc3-a5aa69bb9a3c-istio-envoy\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:11.043115 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:11.042642 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8svxh\" (UniqueName: \"kubernetes.io/projected/0051f207-fce7-4aed-bcc3-a5aa69bb9a3c-kube-api-access-8svxh\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:11.085010 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:11.084972 6720 generic.go:356] "Generic (PLEG): container finished" podID="0051f207-fce7-4aed-bcc3-a5aa69bb9a3c" containerID="76650545cd685ef5bfdeebaaa3a26dbf09d04f9afff648f4fa051bd6185ecf4e" exitCode=0 Oct 30 22:21:11.085199 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:11.085023 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-wc924" Oct 30 22:21:11.085199 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:11.085057 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-wc924" event={"ID":"0051f207-fce7-4aed-bcc3-a5aa69bb9a3c","Type":"ContainerDied","Data":"76650545cd685ef5bfdeebaaa3a26dbf09d04f9afff648f4fa051bd6185ecf4e"} Oct 30 22:21:11.085199 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:11.085119 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-wc924" event={"ID":"0051f207-fce7-4aed-bcc3-a5aa69bb9a3c","Type":"ContainerDied","Data":"eeec6c394d2f953dd5ce92b294c3e39989e06b8de2c7dba8e6396617f53cb123"} Oct 30 22:21:11.085199 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:11.085151 6720 scope.go:122] "RemoveContainer" containerID="76650545cd685ef5bfdeebaaa3a26dbf09d04f9afff648f4fa051bd6185ecf4e" Oct 30 22:21:11.086708 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:11.086686 6720 generic.go:356] "Generic (PLEG): container finished" podID="28b34a7e-4765-46d0-b412-74eb9a1567d9" containerID="2c26c70d220b333ea4dbba0c8cc1cbe4828487c2e752f86fa8d5090f6b391bf1" exitCode=0 Oct 30 22:21:11.086836 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:11.086747 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-n5hzj" Oct 30 22:21:11.086836 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:11.086769 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-n5hzj" event={"ID":"28b34a7e-4765-46d0-b412-74eb9a1567d9","Type":"ContainerDied","Data":"2c26c70d220b333ea4dbba0c8cc1cbe4828487c2e752f86fa8d5090f6b391bf1"} Oct 30 22:21:11.086923 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:11.086835 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-n5hzj" event={"ID":"28b34a7e-4765-46d0-b412-74eb9a1567d9","Type":"ContainerDied","Data":"1c3c7d57beb46ceeca529232994f7f8e4282997238ca5dea897b086fd0fb0fa9"} Oct 30 22:21:11.098452 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:11.098423 6720 scope.go:122] "RemoveContainer" containerID="76650545cd685ef5bfdeebaaa3a26dbf09d04f9afff648f4fa051bd6185ecf4e" Oct 30 22:21:11.098846 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:21:11.098824 6720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76650545cd685ef5bfdeebaaa3a26dbf09d04f9afff648f4fa051bd6185ecf4e\": container with ID starting with 76650545cd685ef5bfdeebaaa3a26dbf09d04f9afff648f4fa051bd6185ecf4e not found: ID does not exist" containerID="76650545cd685ef5bfdeebaaa3a26dbf09d04f9afff648f4fa051bd6185ecf4e" Oct 30 22:21:11.098947 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:11.098861 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76650545cd685ef5bfdeebaaa3a26dbf09d04f9afff648f4fa051bd6185ecf4e"} err="failed to get container status \"76650545cd685ef5bfdeebaaa3a26dbf09d04f9afff648f4fa051bd6185ecf4e\": rpc error: code = NotFound desc = could not find container \"76650545cd685ef5bfdeebaaa3a26dbf09d04f9afff648f4fa051bd6185ecf4e\": container with ID starting with 76650545cd685ef5bfdeebaaa3a26dbf09d04f9afff648f4fa051bd6185ecf4e not found: ID does not exist" Oct 30 22:21:11.098947 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:11.098878 6720 scope.go:122] "RemoveContainer" containerID="2c26c70d220b333ea4dbba0c8cc1cbe4828487c2e752f86fa8d5090f6b391bf1" Oct 30 22:21:11.111309 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:11.111172 6720 scope.go:122] "RemoveContainer" containerID="2c26c70d220b333ea4dbba0c8cc1cbe4828487c2e752f86fa8d5090f6b391bf1" Oct 30 22:21:11.111574 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:21:11.111547 6720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c26c70d220b333ea4dbba0c8cc1cbe4828487c2e752f86fa8d5090f6b391bf1\": container with ID starting with 2c26c70d220b333ea4dbba0c8cc1cbe4828487c2e752f86fa8d5090f6b391bf1 not found: ID does not exist" containerID="2c26c70d220b333ea4dbba0c8cc1cbe4828487c2e752f86fa8d5090f6b391bf1" Oct 30 22:21:11.111657 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:11.111585 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c26c70d220b333ea4dbba0c8cc1cbe4828487c2e752f86fa8d5090f6b391bf1"} err="failed to get container status \"2c26c70d220b333ea4dbba0c8cc1cbe4828487c2e752f86fa8d5090f6b391bf1\": rpc error: code = NotFound desc = could not find container \"2c26c70d220b333ea4dbba0c8cc1cbe4828487c2e752f86fa8d5090f6b391bf1\": container with ID starting with 2c26c70d220b333ea4dbba0c8cc1cbe4828487c2e752f86fa8d5090f6b391bf1 not found: ID does not exist" Oct 30 22:21:11.117682 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:11.117659 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-n5hzj"] Oct 30 22:21:11.122326 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:11.122301 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-n5hzj"] Oct 30 22:21:11.135702 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:11.135673 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-wc924"] Oct 30 22:21:11.141706 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:11.141686 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-wc924"] Oct 30 22:21:11.403655 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:11.402538 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:11.958867 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:11.958828 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0051f207-fce7-4aed-bcc3-a5aa69bb9a3c" path="/var/lib/kubelet/pods/0051f207-fce7-4aed-bcc3-a5aa69bb9a3c/volumes" Oct 30 22:21:11.959311 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:11.959298 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28b34a7e-4765-46d0-b412-74eb9a1567d9" path="/var/lib/kubelet/pods/28b34a7e-4765-46d0-b412-74eb9a1567d9/volumes" Oct 30 22:21:12.034610 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:12.031013 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:12.646413 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:12.644987 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:13.332935 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:13.330829 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:14.014632 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:13.997290 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:14.654140 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:14.654095 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:15.352168 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:15.352126 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:16.033785 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:16.033744 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:16.184660 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:16.184629 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:16.368335 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:16.368300 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:16.783848 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:16.783764 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:16.936194 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:16.936091 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:17.598918 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:17.598882 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:18.346358 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:18.346317 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:19.043150 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:19.043109 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:19.732092 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:19.732045 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:20.460177 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:20.460122 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:21.145438 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:21.133973 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:21.810822 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:21.810781 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:22.440650 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:22.439421 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:23.214016 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:23.213955 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:23.411922 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:23.411888 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:23.576939 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:23.576824 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:23.745170 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:23.745135 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:23.905952 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:23.905911 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:24.085217 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:24.085104 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:24.253058 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:24.252835 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:24.967292 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:24.967255 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:25.132410 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:25.126919 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:26.427762 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:26.427727 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:26.605819 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:26.605779 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:26.900280 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:26.900237 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:27.315108 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:27.314952 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:27.936748 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:27.936667 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:28.679880 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:28.679843 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:29.335200 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:29.335113 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:29.488042 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:29.487981 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:29.989177 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:29.989114 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:30.589209 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:30.589171 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:31.240838 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:31.240801 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:31.640007 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:31.639951 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:31.805357 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:31.805325 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:32.384636 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:32.384590 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:33.138070 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:33.138032 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:33.921790 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:33.921236 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:34.772301 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:34.772260 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:35.525511 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:35.525463 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:36.241318 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:36.241281 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:36.828312 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:36.828274 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:37.532867 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:37.532833 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:38.244137 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:38.244094 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:38.948050 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:38.948009 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:39.563199 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:39.563160 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:40.257993 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:40.257955 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:40.403285 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:21:40.403253 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 22:21:40.968027 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:40.967985 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:41.660047 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:41.660002 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:42.358302 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:42.358232 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:43.061293 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:43.061220 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:43.781701 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:43.781660 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:44.340413 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:44.336584 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:45.240998 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:45.240966 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:46.311309 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:46.311276 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:47.158035 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:47.157992 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:47.877183 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:47.877139 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:48.633302 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:48.633248 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:49.329728 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:49.329671 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:50.067342 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:50.066581 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:50.726358 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:50.726320 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:51.395913 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:51.395684 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:52.438412 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:52.436223 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:53.277109 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:53.277072 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:53.919231 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:53.919130 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:54.675907 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:54.675792 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:55.387869 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:55.385246 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:56.179591 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:56.179551 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:56.808408 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:56.806297 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:57.571962 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:57.571922 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:58.236207 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:58.235012 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:58.973179 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:58.973147 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:21:59.621496 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:21:59.621460 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:00.452061 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:22:00.452024 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:01.246010 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:22:01.245955 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:01.971077 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:22:01.970169 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:02.632227 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:22:02.632115 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:03.348410 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:22:03.332578 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:03.986305 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:22:03.986094 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:04.594266 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:22:04.594167 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:05.235450 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:22:05.235413 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:06.041518 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:22:06.041467 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:06.698098 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:22:06.698053 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:07.324697 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:22:07.324626 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:07.969205 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:22:07.969171 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:08.636412 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:22:08.636278 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:09.251921 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:22:09.251879 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:09.915768 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:22:09.915734 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:10.403670 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:22:10.403630 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 22:22:10.559681 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:22:10.559571 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:11.241414 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:22:11.240941 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:11.817792 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:22:11.817755 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:12.499361 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:22:12.499326 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:13.393331 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:22:13.393275 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:14.005538 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:22:14.005495 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:14.682410 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:22:14.679359 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:15.340936 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:22:15.340904 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:16.164412 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:22:16.163368 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:16.825414 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:22:16.824066 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:17.638322 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:22:17.638277 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:18.396090 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:22:18.396036 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:19.104187 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:22:19.104148 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:19.481253 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:22:19.481173 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:19.816162 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:22:19.816030 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:20.642887 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:22:20.642777 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:21.552739 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:22:21.552702 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:22.276468 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:22:22.276438 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:22.990417 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:22:22.990331 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:23.794369 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:22:23.794319 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:24.511292 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:22:24.511252 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:25.122413 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:22:25.120801 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:25.783930 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:22:25.783878 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:26.648413 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:22:26.648142 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:27.300212 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:22:27.300165 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:28.043437 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:22:28.041546 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:28.634415 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:22:28.629070 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:29.341557 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:22:29.341519 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:29.531290 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:22:29.531257 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:29.977924 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:22:29.977885 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:30.663419 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:22:30.644722 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:31.408314 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:22:31.408279 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:32.044232 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:22:32.044149 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:32.705411 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:22:32.697738 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:33.559433 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:22:33.558204 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:34.327767 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:22:34.327728 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:35.066068 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:22:35.065437 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:35.752155 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:22:35.752120 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:36.662413 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:22:36.659806 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:37.435465 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:22:37.435424 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:38.060416 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:22:38.059193 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:38.816353 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:22:38.816301 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:39.451413 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:22:39.450939 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:40.119543 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:22:40.119458 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:40.404344 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:22:40.404261 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 22:22:40.925792 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:22:40.925752 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:41.525273 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:22:41.516363 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:42.147823 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:22:42.147753 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:42.813123 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:22:42.813090 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:43.470170 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:22:43.469962 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:44.240379 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:22:44.240342 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:44.946736 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:22:44.946670 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:45.548636 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:22:45.548599 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:46.246178 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:22:46.246144 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:46.963072 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:22:46.963038 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:47.761641 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:22:47.761607 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:48.417022 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:22:48.416982 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:49.236297 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:22:49.236262 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:49.959991 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:22:49.959955 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:50.719672 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:22:50.719627 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:51.684778 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:22:51.684723 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:52.341341 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:22:52.341257 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:52.962092 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:22:52.955038 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:53.819998 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:22:53.817888 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:54.538798 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:22:54.538741 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:55.316643 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:22:55.316600 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:56.054079 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:22:56.054040 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:56.745881 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:22:56.745837 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:57.454413 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:22:57.450274 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:58.095417 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:22:58.095133 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:58.759971 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:22:58.759935 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:22:59.599112 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:22:59.599073 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:00.249408 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:00.238762 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:00.926532 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:00.926497 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:01.645415 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:01.643619 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:02.451516 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:02.451476 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:03.323934 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:03.317617 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:04.143096 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:04.137615 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:04.910653 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:04.910606 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:05.632471 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:05.632433 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:06.431587 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:06.431557 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:07.165739 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:07.147774 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:07.866409 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:07.864815 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:08.465407 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:08.464506 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:09.191336 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:09.191298 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:09.851597 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:09.851560 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:10.404160 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:10.404113 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9"] Oct 30 22:23:10.404477 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:10.404451 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9" podUID="13913563-7222-407c-98b1-5c1a17d10b82" containerName="istio-proxy" containerID="cri-o://13d0f98a7a026a629eb0f6f0c23d0fbd558e7ca662aa0a669e51da5a049520a4" gracePeriod=30 Oct 30 22:23:10.406891 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:23:10.406858 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 22:23:10.447905 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:10.447874 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:11.144727 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:11.144687 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:11.844151 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:11.844115 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:12.599172 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:12.599138 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:13.232418 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:13.226914 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:13.898935 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:13.898897 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:14.286825 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:14.286739 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-2tv9m"] Oct 30 22:23:14.286970 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:14.286953 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-2tv9m" podUID="0bd82fe0-1de3-40c8-b825-51eff2ae3386" containerName="istio-proxy" containerID="cri-o://baba776f8ac85fccfd4545d6dbdbe219f96bee72ac4a9b7dd9da08595218b497" gracePeriod=30 Oct 30 22:23:14.832152 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:14.832105 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:15.527263 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:15.527130 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:15.582619 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:15.579028 6720 patch_prober.go:29] interesting pod/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 22:23:15.582619 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:15.579114 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9" podUID="13913563-7222-407c-98b1-5c1a17d10b82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 22:23:15.773465 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:15.773440 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9" Oct 30 22:23:15.853736 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:15.853703 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13913563-7222-407c-98b1-5c1a17d10b82-istio-token\" (UniqueName: \"kubernetes.io/projected/13913563-7222-407c-98b1-5c1a17d10b82-istio-token\") pod \"13913563-7222-407c-98b1-5c1a17d10b82\" (UID: \"13913563-7222-407c-98b1-5c1a17d10b82\") " Oct 30 22:23:15.853994 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:15.853971 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13913563-7222-407c-98b1-5c1a17d10b82-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13913563-7222-407c-98b1-5c1a17d10b82-istio-envoy\") pod \"13913563-7222-407c-98b1-5c1a17d10b82\" (UID: \"13913563-7222-407c-98b1-5c1a17d10b82\") " Oct 30 22:23:15.854133 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:15.854036 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13913563-7222-407c-98b1-5c1a17d10b82-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13913563-7222-407c-98b1-5c1a17d10b82-workload-certs\") pod \"13913563-7222-407c-98b1-5c1a17d10b82\" (UID: \"13913563-7222-407c-98b1-5c1a17d10b82\") " Oct 30 22:23:15.854133 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:15.854081 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/13913563-7222-407c-98b1-5c1a17d10b82-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13913563-7222-407c-98b1-5c1a17d10b82-istio-podinfo\") pod \"13913563-7222-407c-98b1-5c1a17d10b82\" (UID: \"13913563-7222-407c-98b1-5c1a17d10b82\") " Oct 30 22:23:15.854133 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:15.854124 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13913563-7222-407c-98b1-5c1a17d10b82-kube-api-access-sg84w\" (UniqueName: \"kubernetes.io/projected/13913563-7222-407c-98b1-5c1a17d10b82-kube-api-access-sg84w\") pod \"13913563-7222-407c-98b1-5c1a17d10b82\" (UID: \"13913563-7222-407c-98b1-5c1a17d10b82\") " Oct 30 22:23:15.854347 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:15.854310 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13913563-7222-407c-98b1-5c1a17d10b82-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13913563-7222-407c-98b1-5c1a17d10b82-credential-socket\") pod \"13913563-7222-407c-98b1-5c1a17d10b82\" (UID: \"13913563-7222-407c-98b1-5c1a17d10b82\") " Oct 30 22:23:15.854423 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:15.854364 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13913563-7222-407c-98b1-5c1a17d10b82-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13913563-7222-407c-98b1-5c1a17d10b82-istio-data\") pod \"13913563-7222-407c-98b1-5c1a17d10b82\" (UID: \"13913563-7222-407c-98b1-5c1a17d10b82\") " Oct 30 22:23:15.854772 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:15.854475 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13913563-7222-407c-98b1-5c1a17d10b82-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13913563-7222-407c-98b1-5c1a17d10b82-workload-socket\") pod \"13913563-7222-407c-98b1-5c1a17d10b82\" (UID: \"13913563-7222-407c-98b1-5c1a17d10b82\") " Oct 30 22:23:15.854772 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:15.854513 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13913563-7222-407c-98b1-5c1a17d10b82-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13913563-7222-407c-98b1-5c1a17d10b82-istiod-ca-cert\") pod \"13913563-7222-407c-98b1-5c1a17d10b82\" (UID: \"13913563-7222-407c-98b1-5c1a17d10b82\") " Oct 30 22:23:15.855776 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:15.855751 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13913563-7222-407c-98b1-5c1a17d10b82-workload-socket" pod "13913563-7222-407c-98b1-5c1a17d10b82" (UID: "13913563-7222-407c-98b1-5c1a17d10b82"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:15.855899 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:15.855784 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13913563-7222-407c-98b1-5c1a17d10b82-istiod-ca-cert" pod "13913563-7222-407c-98b1-5c1a17d10b82" (UID: "13913563-7222-407c-98b1-5c1a17d10b82"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:23:15.856020 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:15.855997 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13913563-7222-407c-98b1-5c1a17d10b82-credential-socket" pod "13913563-7222-407c-98b1-5c1a17d10b82" (UID: "13913563-7222-407c-98b1-5c1a17d10b82"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:15.856161 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:15.856060 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13913563-7222-407c-98b1-5c1a17d10b82-workload-certs" pod "13913563-7222-407c-98b1-5c1a17d10b82" (UID: "13913563-7222-407c-98b1-5c1a17d10b82"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:15.856281 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:15.856108 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13913563-7222-407c-98b1-5c1a17d10b82-istio-data" pod "13913563-7222-407c-98b1-5c1a17d10b82" (UID: "13913563-7222-407c-98b1-5c1a17d10b82"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:15.856715 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:15.856687 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13913563-7222-407c-98b1-5c1a17d10b82-istio-token" pod "13913563-7222-407c-98b1-5c1a17d10b82" (UID: "13913563-7222-407c-98b1-5c1a17d10b82"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:15.857119 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:15.857098 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13913563-7222-407c-98b1-5c1a17d10b82-istio-envoy" pod "13913563-7222-407c-98b1-5c1a17d10b82" (UID: "13913563-7222-407c-98b1-5c1a17d10b82"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:15.858450 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:15.858375 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13913563-7222-407c-98b1-5c1a17d10b82-kube-api-access-sg84w" pod "13913563-7222-407c-98b1-5c1a17d10b82" (UID: "13913563-7222-407c-98b1-5c1a17d10b82"). InnerVolumeSpecName "kube-api-access-sg84w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:15.859188 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:15.859167 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/13913563-7222-407c-98b1-5c1a17d10b82-istio-podinfo" pod "13913563-7222-407c-98b1-5c1a17d10b82" (UID: "13913563-7222-407c-98b1-5c1a17d10b82"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:23:15.956339 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:15.956305 6720 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13913563-7222-407c-98b1-5c1a17d10b82-istio-data\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:15.956339 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:15.956334 6720 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13913563-7222-407c-98b1-5c1a17d10b82-workload-socket\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:15.956339 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:15.956345 6720 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13913563-7222-407c-98b1-5c1a17d10b82-istiod-ca-cert\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:15.956609 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:15.956355 6720 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13913563-7222-407c-98b1-5c1a17d10b82-istio-token\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:15.956609 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:15.956363 6720 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13913563-7222-407c-98b1-5c1a17d10b82-istio-envoy\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:15.956609 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:15.956373 6720 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13913563-7222-407c-98b1-5c1a17d10b82-workload-certs\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:15.956609 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:15.956398 6720 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13913563-7222-407c-98b1-5c1a17d10b82-istio-podinfo\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:15.956609 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:15.956412 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sg84w\" (UniqueName: \"kubernetes.io/projected/13913563-7222-407c-98b1-5c1a17d10b82-kube-api-access-sg84w\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:15.956609 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:15.956425 6720 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13913563-7222-407c-98b1-5c1a17d10b82-credential-socket\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:16.209919 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:16.209877 6720 generic.go:356] "Generic (PLEG): container finished" podID="13913563-7222-407c-98b1-5c1a17d10b82" containerID="13d0f98a7a026a629eb0f6f0c23d0fbd558e7ca662aa0a669e51da5a049520a4" exitCode=0 Oct 30 22:23:16.210113 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:16.209926 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9" Oct 30 22:23:16.210113 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:16.209952 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9" event={"ID":"13913563-7222-407c-98b1-5c1a17d10b82","Type":"ContainerDied","Data":"13d0f98a7a026a629eb0f6f0c23d0fbd558e7ca662aa0a669e51da5a049520a4"} Oct 30 22:23:16.210113 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:16.210017 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9" event={"ID":"13913563-7222-407c-98b1-5c1a17d10b82","Type":"ContainerDied","Data":"38b4b9210ac7ec93935a3efa37cde37887e7dfb3384f4e6f86a6e61dc0ed35e0"} Oct 30 22:23:16.210113 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:16.210048 6720 scope.go:122] "RemoveContainer" containerID="13d0f98a7a026a629eb0f6f0c23d0fbd558e7ca662aa0a669e51da5a049520a4" Oct 30 22:23:16.223483 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:16.223458 6720 scope.go:122] "RemoveContainer" containerID="13d0f98a7a026a629eb0f6f0c23d0fbd558e7ca662aa0a669e51da5a049520a4" Oct 30 22:23:16.223894 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:23:16.223854 6720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13d0f98a7a026a629eb0f6f0c23d0fbd558e7ca662aa0a669e51da5a049520a4\": container with ID starting with 13d0f98a7a026a629eb0f6f0c23d0fbd558e7ca662aa0a669e51da5a049520a4 not found: ID does not exist" containerID="13d0f98a7a026a629eb0f6f0c23d0fbd558e7ca662aa0a669e51da5a049520a4" Oct 30 22:23:16.224037 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:16.223904 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13d0f98a7a026a629eb0f6f0c23d0fbd558e7ca662aa0a669e51da5a049520a4"} err="failed to get container status \"13d0f98a7a026a629eb0f6f0c23d0fbd558e7ca662aa0a669e51da5a049520a4\": rpc error: code = NotFound desc = could not find container \"13d0f98a7a026a629eb0f6f0c23d0fbd558e7ca662aa0a669e51da5a049520a4\": container with ID starting with 13d0f98a7a026a629eb0f6f0c23d0fbd558e7ca662aa0a669e51da5a049520a4 not found: ID does not exist" Oct 30 22:23:16.224257 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:16.224232 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:16.234884 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:16.234851 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9"] Oct 30 22:23:16.236888 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:16.236862 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-pvgb9"] Oct 30 22:23:16.938368 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:16.938335 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:17.778706 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:17.778034 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:17.958750 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:17.958707 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13913563-7222-407c-98b1-5c1a17d10b82" path="/var/lib/kubelet/pods/13913563-7222-407c-98b1-5c1a17d10b82/volumes" Oct 30 22:23:18.350434 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:18.349252 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:18.591583 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:18.591549 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:19.039575 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:19.039539 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:19.573584 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:19.573558 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-2tv9m" Oct 30 22:23:19.648085 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:19.648037 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:19.737405 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:19.730136 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bd82fe0-1de3-40c8-b825-51eff2ae3386-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bd82fe0-1de3-40c8-b825-51eff2ae3386-istio-data\") pod \"0bd82fe0-1de3-40c8-b825-51eff2ae3386\" (UID: \"0bd82fe0-1de3-40c8-b825-51eff2ae3386\") " Oct 30 22:23:19.737405 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:19.730222 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bd82fe0-1de3-40c8-b825-51eff2ae3386-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bd82fe0-1de3-40c8-b825-51eff2ae3386-istio-envoy\") pod \"0bd82fe0-1de3-40c8-b825-51eff2ae3386\" (UID: \"0bd82fe0-1de3-40c8-b825-51eff2ae3386\") " Oct 30 22:23:19.737405 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:19.730293 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bd82fe0-1de3-40c8-b825-51eff2ae3386-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bd82fe0-1de3-40c8-b825-51eff2ae3386-workload-certs\") pod \"0bd82fe0-1de3-40c8-b825-51eff2ae3386\" (UID: \"0bd82fe0-1de3-40c8-b825-51eff2ae3386\") " Oct 30 22:23:19.737405 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:19.730358 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bd82fe0-1de3-40c8-b825-51eff2ae3386-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bd82fe0-1de3-40c8-b825-51eff2ae3386-istiod-ca-cert\") pod \"0bd82fe0-1de3-40c8-b825-51eff2ae3386\" (UID: \"0bd82fe0-1de3-40c8-b825-51eff2ae3386\") " Oct 30 22:23:19.737405 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:19.730663 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bd82fe0-1de3-40c8-b825-51eff2ae3386-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bd82fe0-1de3-40c8-b825-51eff2ae3386-workload-socket\") pod \"0bd82fe0-1de3-40c8-b825-51eff2ae3386\" (UID: \"0bd82fe0-1de3-40c8-b825-51eff2ae3386\") " Oct 30 22:23:19.737405 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:19.730734 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bd82fe0-1de3-40c8-b825-51eff2ae3386-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bd82fe0-1de3-40c8-b825-51eff2ae3386-credential-socket\") pod \"0bd82fe0-1de3-40c8-b825-51eff2ae3386\" (UID: \"0bd82fe0-1de3-40c8-b825-51eff2ae3386\") " Oct 30 22:23:19.737405 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:19.730776 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bd82fe0-1de3-40c8-b825-51eff2ae3386-kube-api-access-zxnsd\" (UniqueName: \"kubernetes.io/projected/0bd82fe0-1de3-40c8-b825-51eff2ae3386-kube-api-access-zxnsd\") pod \"0bd82fe0-1de3-40c8-b825-51eff2ae3386\" (UID: \"0bd82fe0-1de3-40c8-b825-51eff2ae3386\") " Oct 30 22:23:19.737405 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:19.730902 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0bd82fe0-1de3-40c8-b825-51eff2ae3386-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bd82fe0-1de3-40c8-b825-51eff2ae3386-istio-podinfo\") pod \"0bd82fe0-1de3-40c8-b825-51eff2ae3386\" (UID: \"0bd82fe0-1de3-40c8-b825-51eff2ae3386\") " Oct 30 22:23:19.737405 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:19.731001 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bd82fe0-1de3-40c8-b825-51eff2ae3386-istio-token\" (UniqueName: \"kubernetes.io/projected/0bd82fe0-1de3-40c8-b825-51eff2ae3386-istio-token\") pod \"0bd82fe0-1de3-40c8-b825-51eff2ae3386\" (UID: \"0bd82fe0-1de3-40c8-b825-51eff2ae3386\") " Oct 30 22:23:19.747686 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:19.739122 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bd82fe0-1de3-40c8-b825-51eff2ae3386-istiod-ca-cert" pod "0bd82fe0-1de3-40c8-b825-51eff2ae3386" (UID: "0bd82fe0-1de3-40c8-b825-51eff2ae3386"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:23:19.747686 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:19.743483 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bd82fe0-1de3-40c8-b825-51eff2ae3386-credential-socket" pod "0bd82fe0-1de3-40c8-b825-51eff2ae3386" (UID: "0bd82fe0-1de3-40c8-b825-51eff2ae3386"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:19.747686 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:19.744049 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bd82fe0-1de3-40c8-b825-51eff2ae3386-workload-certs" pod "0bd82fe0-1de3-40c8-b825-51eff2ae3386" (UID: "0bd82fe0-1de3-40c8-b825-51eff2ae3386"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:19.749508 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:19.749471 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bd82fe0-1de3-40c8-b825-51eff2ae3386-istio-data" pod "0bd82fe0-1de3-40c8-b825-51eff2ae3386" (UID: "0bd82fe0-1de3-40c8-b825-51eff2ae3386"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:19.754994 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:19.754962 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bd82fe0-1de3-40c8-b825-51eff2ae3386-workload-socket" pod "0bd82fe0-1de3-40c8-b825-51eff2ae3386" (UID: "0bd82fe0-1de3-40c8-b825-51eff2ae3386"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:19.783512 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:19.783473 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bd82fe0-1de3-40c8-b825-51eff2ae3386-istio-envoy" pod "0bd82fe0-1de3-40c8-b825-51eff2ae3386" (UID: "0bd82fe0-1de3-40c8-b825-51eff2ae3386"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:19.783783 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:19.783765 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0bd82fe0-1de3-40c8-b825-51eff2ae3386-istio-podinfo" pod "0bd82fe0-1de3-40c8-b825-51eff2ae3386" (UID: "0bd82fe0-1de3-40c8-b825-51eff2ae3386"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:23:19.786213 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:19.786185 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bd82fe0-1de3-40c8-b825-51eff2ae3386-kube-api-access-zxnsd" pod "0bd82fe0-1de3-40c8-b825-51eff2ae3386" (UID: "0bd82fe0-1de3-40c8-b825-51eff2ae3386"). InnerVolumeSpecName "kube-api-access-zxnsd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:19.789489 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:19.788806 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bd82fe0-1de3-40c8-b825-51eff2ae3386-istio-token" pod "0bd82fe0-1de3-40c8-b825-51eff2ae3386" (UID: "0bd82fe0-1de3-40c8-b825-51eff2ae3386"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:19.838967 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:19.838890 6720 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bd82fe0-1de3-40c8-b825-51eff2ae3386-istio-token\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:19.838967 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:19.838920 6720 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bd82fe0-1de3-40c8-b825-51eff2ae3386-istio-data\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:19.838967 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:19.838929 6720 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bd82fe0-1de3-40c8-b825-51eff2ae3386-istio-envoy\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:19.838967 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:19.838938 6720 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bd82fe0-1de3-40c8-b825-51eff2ae3386-workload-certs\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:19.838967 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:19.838947 6720 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bd82fe0-1de3-40c8-b825-51eff2ae3386-istiod-ca-cert\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:19.838967 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:19.838956 6720 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bd82fe0-1de3-40c8-b825-51eff2ae3386-workload-socket\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:19.838967 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:19.838964 6720 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bd82fe0-1de3-40c8-b825-51eff2ae3386-credential-socket\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:19.839293 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:19.838974 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxnsd\" (UniqueName: \"kubernetes.io/projected/0bd82fe0-1de3-40c8-b825-51eff2ae3386-kube-api-access-zxnsd\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:19.839293 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:19.838984 6720 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bd82fe0-1de3-40c8-b825-51eff2ae3386-istio-podinfo\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:20.238068 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:20.238034 6720 generic.go:356] "Generic (PLEG): container finished" podID="0bd82fe0-1de3-40c8-b825-51eff2ae3386" containerID="baba776f8ac85fccfd4545d6dbdbe219f96bee72ac4a9b7dd9da08595218b497" exitCode=0 Oct 30 22:23:20.238549 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:20.238087 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-2tv9m" Oct 30 22:23:20.238549 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:20.238109 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-2tv9m" event={"ID":"0bd82fe0-1de3-40c8-b825-51eff2ae3386","Type":"ContainerDied","Data":"baba776f8ac85fccfd4545d6dbdbe219f96bee72ac4a9b7dd9da08595218b497"} Oct 30 22:23:20.238549 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:20.238161 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-2tv9m" event={"ID":"0bd82fe0-1de3-40c8-b825-51eff2ae3386","Type":"ContainerDied","Data":"e00099c03bc7cae9fd4819887f91fbb82e71daf43a7b3ec0d0076abba3fddee3"} Oct 30 22:23:20.238549 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:20.238183 6720 scope.go:122] "RemoveContainer" containerID="baba776f8ac85fccfd4545d6dbdbe219f96bee72ac4a9b7dd9da08595218b497" Oct 30 22:23:20.251567 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:20.251492 6720 scope.go:122] "RemoveContainer" containerID="baba776f8ac85fccfd4545d6dbdbe219f96bee72ac4a9b7dd9da08595218b497" Oct 30 22:23:20.251893 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:23:20.251872 6720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baba776f8ac85fccfd4545d6dbdbe219f96bee72ac4a9b7dd9da08595218b497\": container with ID starting with baba776f8ac85fccfd4545d6dbdbe219f96bee72ac4a9b7dd9da08595218b497 not found: ID does not exist" containerID="baba776f8ac85fccfd4545d6dbdbe219f96bee72ac4a9b7dd9da08595218b497" Oct 30 22:23:20.251986 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:20.251904 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baba776f8ac85fccfd4545d6dbdbe219f96bee72ac4a9b7dd9da08595218b497"} err="failed to get container status \"baba776f8ac85fccfd4545d6dbdbe219f96bee72ac4a9b7dd9da08595218b497\": rpc error: code = NotFound desc = could not find container \"baba776f8ac85fccfd4545d6dbdbe219f96bee72ac4a9b7dd9da08595218b497\": container with ID starting with baba776f8ac85fccfd4545d6dbdbe219f96bee72ac4a9b7dd9da08595218b497 not found: ID does not exist" Oct 30 22:23:20.259537 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:20.259510 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-2tv9m"] Oct 30 22:23:20.266410 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:20.266361 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-2tv9m"] Oct 30 22:23:20.422641 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:20.422601 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:21.143742 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:21.143649 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:21.928416 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:21.923966 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:21.968531 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:21.968502 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bd82fe0-1de3-40c8-b825-51eff2ae3386" path="/var/lib/kubelet/pods/0bd82fe0-1de3-40c8-b825-51eff2ae3386/volumes" Oct 30 22:23:22.554556 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:22.554525 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:23.239445 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:23.239311 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:23.824846 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:23.824781 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:24.513064 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:24.513023 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:25.155642 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:25.155606 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:25.941258 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:25.941222 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:26.721599 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:26.721562 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:27.522858 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:27.522751 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:28.229727 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:28.229679 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:28.926858 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:28.926764 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:29.592161 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:29.592116 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:29.733124 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:29.733072 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:30.239120 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:30.239042 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:30.930786 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:30.930744 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:31.627863 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:31.627828 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:32.221203 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:32.221175 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:32.843572 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:32.843509 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:33.516206 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:33.516121 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:34.147412 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:34.142060 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:34.935603 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:34.935566 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:35.602280 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:35.602222 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:35.934446 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:35.934405 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-l6qmw"] Oct 30 22:23:35.934674 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:35.934638 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-l6qmw" podUID="ad5f36c9-9a30-43b1-834b-3ab9c458bdb0" containerName="istio-proxy" containerID="cri-o://95c8c2ee943f94b7c8753a117968f13599665423a367e74b2377d2b3c9127829" gracePeriod=30 Oct 30 22:23:36.552652 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:36.552605 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:36.717077 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:36.717045 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:36.884423 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:36.881372 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:37.593441 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:37.593318 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:38.323875 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:38.323843 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:38.474521 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:38.474484 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:38.620902 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:38.620868 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:38.774283 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:38.774244 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:39.043947 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:39.043854 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:39.206811 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:39.206782 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:39.358033 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:39.357991 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:39.535680 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:39.535648 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:39.843966 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:39.843819 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:40.403812 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:23:40.403775 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 22:23:40.745263 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:40.745179 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:40.960408 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:40.957623 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:41.360773 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:41.360743 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-l6qmw" Oct 30 22:23:41.425885 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:41.425847 6720 generic.go:356] "Generic (PLEG): container finished" podID="ad5f36c9-9a30-43b1-834b-3ab9c458bdb0" containerID="95c8c2ee943f94b7c8753a117968f13599665423a367e74b2377d2b3c9127829" exitCode=0 Oct 30 22:23:41.426361 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:41.425901 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-l6qmw" Oct 30 22:23:41.426361 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:41.425910 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-l6qmw" event={"ID":"ad5f36c9-9a30-43b1-834b-3ab9c458bdb0","Type":"ContainerDied","Data":"95c8c2ee943f94b7c8753a117968f13599665423a367e74b2377d2b3c9127829"} Oct 30 22:23:41.426361 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:41.425986 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-l6qmw" event={"ID":"ad5f36c9-9a30-43b1-834b-3ab9c458bdb0","Type":"ContainerDied","Data":"11b2aa5a1383bac58135b87cf09e69a1cfd9d50ba6ae6aa0730ade8460ca05b4"} Oct 30 22:23:41.426361 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:41.426023 6720 scope.go:122] "RemoveContainer" containerID="95c8c2ee943f94b7c8753a117968f13599665423a367e74b2377d2b3c9127829" Oct 30 22:23:41.438573 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:41.438549 6720 scope.go:122] "RemoveContainer" containerID="95c8c2ee943f94b7c8753a117968f13599665423a367e74b2377d2b3c9127829" Oct 30 22:23:41.438917 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:23:41.438891 6720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95c8c2ee943f94b7c8753a117968f13599665423a367e74b2377d2b3c9127829\": container with ID starting with 95c8c2ee943f94b7c8753a117968f13599665423a367e74b2377d2b3c9127829 not found: ID does not exist" containerID="95c8c2ee943f94b7c8753a117968f13599665423a367e74b2377d2b3c9127829" Oct 30 22:23:41.439005 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:41.438933 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95c8c2ee943f94b7c8753a117968f13599665423a367e74b2377d2b3c9127829"} err="failed to get container status \"95c8c2ee943f94b7c8753a117968f13599665423a367e74b2377d2b3c9127829\": rpc error: code = NotFound desc = could not find container \"95c8c2ee943f94b7c8753a117968f13599665423a367e74b2377d2b3c9127829\": container with ID starting with 95c8c2ee943f94b7c8753a117968f13599665423a367e74b2377d2b3c9127829 not found: ID does not exist" Oct 30 22:23:41.492184 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:41.492142 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad5f36c9-9a30-43b1-834b-3ab9c458bdb0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad5f36c9-9a30-43b1-834b-3ab9c458bdb0-workload-socket\") pod \"ad5f36c9-9a30-43b1-834b-3ab9c458bdb0\" (UID: \"ad5f36c9-9a30-43b1-834b-3ab9c458bdb0\") " Oct 30 22:23:41.492371 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:41.492225 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad5f36c9-9a30-43b1-834b-3ab9c458bdb0-istio-token\" (UniqueName: \"kubernetes.io/projected/ad5f36c9-9a30-43b1-834b-3ab9c458bdb0-istio-token\") pod \"ad5f36c9-9a30-43b1-834b-3ab9c458bdb0\" (UID: \"ad5f36c9-9a30-43b1-834b-3ab9c458bdb0\") " Oct 30 22:23:41.492462 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:41.492371 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad5f36c9-9a30-43b1-834b-3ab9c458bdb0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad5f36c9-9a30-43b1-834b-3ab9c458bdb0-credential-socket\") pod \"ad5f36c9-9a30-43b1-834b-3ab9c458bdb0\" (UID: \"ad5f36c9-9a30-43b1-834b-3ab9c458bdb0\") " Oct 30 22:23:41.492462 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:41.492420 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad5f36c9-9a30-43b1-834b-3ab9c458bdb0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad5f36c9-9a30-43b1-834b-3ab9c458bdb0-istiod-ca-cert\") pod \"ad5f36c9-9a30-43b1-834b-3ab9c458bdb0\" (UID: \"ad5f36c9-9a30-43b1-834b-3ab9c458bdb0\") " Oct 30 22:23:41.492462 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:41.492423 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad5f36c9-9a30-43b1-834b-3ab9c458bdb0-workload-socket" pod "ad5f36c9-9a30-43b1-834b-3ab9c458bdb0" (UID: "ad5f36c9-9a30-43b1-834b-3ab9c458bdb0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:41.492631 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:41.492557 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad5f36c9-9a30-43b1-834b-3ab9c458bdb0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad5f36c9-9a30-43b1-834b-3ab9c458bdb0-istio-data\") pod \"ad5f36c9-9a30-43b1-834b-3ab9c458bdb0\" (UID: \"ad5f36c9-9a30-43b1-834b-3ab9c458bdb0\") " Oct 30 22:23:41.492689 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:41.492625 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad5f36c9-9a30-43b1-834b-3ab9c458bdb0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad5f36c9-9a30-43b1-834b-3ab9c458bdb0-workload-certs\") pod \"ad5f36c9-9a30-43b1-834b-3ab9c458bdb0\" (UID: \"ad5f36c9-9a30-43b1-834b-3ab9c458bdb0\") " Oct 30 22:23:41.492689 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:41.492659 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad5f36c9-9a30-43b1-834b-3ab9c458bdb0-credential-socket" pod "ad5f36c9-9a30-43b1-834b-3ab9c458bdb0" (UID: "ad5f36c9-9a30-43b1-834b-3ab9c458bdb0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:41.492689 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:41.492673 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad5f36c9-9a30-43b1-834b-3ab9c458bdb0-kube-api-access-977js\" (UniqueName: \"kubernetes.io/projected/ad5f36c9-9a30-43b1-834b-3ab9c458bdb0-kube-api-access-977js\") pod \"ad5f36c9-9a30-43b1-834b-3ab9c458bdb0\" (UID: \"ad5f36c9-9a30-43b1-834b-3ab9c458bdb0\") " Oct 30 22:23:41.492853 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:41.492703 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ad5f36c9-9a30-43b1-834b-3ab9c458bdb0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad5f36c9-9a30-43b1-834b-3ab9c458bdb0-istio-podinfo\") pod \"ad5f36c9-9a30-43b1-834b-3ab9c458bdb0\" (UID: \"ad5f36c9-9a30-43b1-834b-3ab9c458bdb0\") " Oct 30 22:23:41.492853 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:41.492739 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad5f36c9-9a30-43b1-834b-3ab9c458bdb0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad5f36c9-9a30-43b1-834b-3ab9c458bdb0-istio-envoy\") pod \"ad5f36c9-9a30-43b1-834b-3ab9c458bdb0\" (UID: \"ad5f36c9-9a30-43b1-834b-3ab9c458bdb0\") " Oct 30 22:23:41.492853 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:41.492841 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad5f36c9-9a30-43b1-834b-3ab9c458bdb0-istiod-ca-cert" pod "ad5f36c9-9a30-43b1-834b-3ab9c458bdb0" (UID: "ad5f36c9-9a30-43b1-834b-3ab9c458bdb0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:23:41.493274 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:41.493254 6720 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad5f36c9-9a30-43b1-834b-3ab9c458bdb0-workload-socket\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:41.493370 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:41.493280 6720 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad5f36c9-9a30-43b1-834b-3ab9c458bdb0-credential-socket\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:41.493370 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:41.493295 6720 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad5f36c9-9a30-43b1-834b-3ab9c458bdb0-istiod-ca-cert\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:41.493644 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:41.493621 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad5f36c9-9a30-43b1-834b-3ab9c458bdb0-istio-data" pod "ad5f36c9-9a30-43b1-834b-3ab9c458bdb0" (UID: "ad5f36c9-9a30-43b1-834b-3ab9c458bdb0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:41.493719 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:41.493637 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad5f36c9-9a30-43b1-834b-3ab9c458bdb0-workload-certs" pod "ad5f36c9-9a30-43b1-834b-3ab9c458bdb0" (UID: "ad5f36c9-9a30-43b1-834b-3ab9c458bdb0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:41.494964 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:41.494933 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad5f36c9-9a30-43b1-834b-3ab9c458bdb0-istio-token" pod "ad5f36c9-9a30-43b1-834b-3ab9c458bdb0" (UID: "ad5f36c9-9a30-43b1-834b-3ab9c458bdb0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:41.496130 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:41.496101 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad5f36c9-9a30-43b1-834b-3ab9c458bdb0-kube-api-access-977js" pod "ad5f36c9-9a30-43b1-834b-3ab9c458bdb0" (UID: "ad5f36c9-9a30-43b1-834b-3ab9c458bdb0"). InnerVolumeSpecName "kube-api-access-977js". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:23:41.496224 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:41.496165 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ad5f36c9-9a30-43b1-834b-3ab9c458bdb0-istio-podinfo" pod "ad5f36c9-9a30-43b1-834b-3ab9c458bdb0" (UID: "ad5f36c9-9a30-43b1-834b-3ab9c458bdb0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:23:41.496278 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:41.496227 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad5f36c9-9a30-43b1-834b-3ab9c458bdb0-istio-envoy" pod "ad5f36c9-9a30-43b1-834b-3ab9c458bdb0" (UID: "ad5f36c9-9a30-43b1-834b-3ab9c458bdb0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:23:41.600459 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:41.597496 6720 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad5f36c9-9a30-43b1-834b-3ab9c458bdb0-istio-data\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:41.600459 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:41.597538 6720 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad5f36c9-9a30-43b1-834b-3ab9c458bdb0-workload-certs\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:41.600459 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:41.597555 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-977js\" (UniqueName: \"kubernetes.io/projected/ad5f36c9-9a30-43b1-834b-3ab9c458bdb0-kube-api-access-977js\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:41.600459 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:41.597570 6720 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad5f36c9-9a30-43b1-834b-3ab9c458bdb0-istio-podinfo\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:41.600459 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:41.597587 6720 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad5f36c9-9a30-43b1-834b-3ab9c458bdb0-istio-envoy\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:41.600459 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:41.597601 6720 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad5f36c9-9a30-43b1-834b-3ab9c458bdb0-istio-token\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:23:41.755910 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:41.755550 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-l6qmw"] Oct 30 22:23:41.759988 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:41.759951 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-l6qmw"] Oct 30 22:23:41.958634 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:41.958593 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad5f36c9-9a30-43b1-834b-3ab9c458bdb0" path="/var/lib/kubelet/pods/ad5f36c9-9a30-43b1-834b-3ab9c458bdb0/volumes" Oct 30 22:23:42.111440 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:42.109237 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:42.944455 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:42.943953 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:43.803163 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:43.803127 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:44.494539 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:44.494497 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:45.221091 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:45.221021 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:46.045214 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:46.045155 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:46.645777 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:46.641730 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:47.427449 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:47.427411 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:48.135649 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:48.135615 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:48.587231 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:48.587141 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:48.738139 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:48.738093 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:49.364805 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:49.364754 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:50.358003 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:50.357961 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:50.542240 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:50.542201 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:50.757166 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:50.757058 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:51.744851 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:51.744814 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:51.914553 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:51.913456 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:52.201692 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:52.201657 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:52.706853 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:52.706803 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:53.349410 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:53.348888 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:54.117458 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:54.117418 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:54.777556 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:54.777528 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:55.460365 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:55.460332 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:56.347794 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:56.347750 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:56.958180 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:56.958142 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:57.719023 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:57.718984 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:58.658411 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:58.652308 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:23:59.343779 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:23:59.343746 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:00.642869 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:00.642821 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:01.474996 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:01.474943 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:01.729253 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:01.729164 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:01.897014 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:01.896975 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:03.096620 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:03.096561 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:03.848466 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:03.841955 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:04.635410 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:04.633070 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:05.245108 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:05.245051 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:06.067100 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:06.067060 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:06.793803 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:06.793768 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:07.419069 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:07.419033 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:08.044991 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:08.044954 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:08.760906 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:08.760876 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:09.339626 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:09.339572 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:09.493987 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:09.493958 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:09.643412 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:09.643355 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:10.405294 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:24:10.405255 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 22:24:10.509966 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:10.509874 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:11.170021 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:11.169980 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:11.328688 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:11.328654 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:11.474074 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:11.473975 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:12.268948 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:12.268914 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:12.950444 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:12.950374 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:13.597054 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:13.596810 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:14.309921 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:14.309836 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:15.355665 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:15.355631 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:15.734269 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:15.733727 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:16.365273 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:16.364965 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:17.131873 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:17.131820 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:17.844914 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:17.844875 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:18.486594 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:18.486556 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:19.137127 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:19.137076 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:19.867869 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:19.863482 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:20.601582 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:20.601506 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:21.036850 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:21.036807 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-7pjjd"] Oct 30 22:24:21.037538 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:21.037517 6720 cpu_manager.go:414] "containerMap: removing container" podUID="de3025de-693a-4527-a04e-c9dd6d3caf89" containerName="istio-proxy" Oct 30 22:24:21.037538 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:21.037539 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de3025de-693a-4527-a04e-c9dd6d3caf89" containerName="istio-proxy" Oct 30 22:24:21.037724 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:21.037554 6720 cpu_manager.go:414] "containerMap: removing container" podUID="28b34a7e-4765-46d0-b412-74eb9a1567d9" containerName="istio-proxy" Oct 30 22:24:21.037724 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:21.037562 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28b34a7e-4765-46d0-b412-74eb9a1567d9" containerName="istio-proxy" Oct 30 22:24:21.037724 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:21.037584 6720 cpu_manager.go:414] "containerMap: removing container" podUID="5359ba6b-7deb-4055-9069-1e59a9c4e392" containerName="istio-proxy" Oct 30 22:24:21.037724 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:21.037593 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5359ba6b-7deb-4055-9069-1e59a9c4e392" containerName="istio-proxy" Oct 30 22:24:21.037724 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:21.037623 6720 cpu_manager.go:414] "containerMap: removing container" podUID="0051f207-fce7-4aed-bcc3-a5aa69bb9a3c" containerName="istio-proxy" Oct 30 22:24:21.037724 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:21.037631 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0051f207-fce7-4aed-bcc3-a5aa69bb9a3c" containerName="istio-proxy" Oct 30 22:24:21.037724 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:21.037650 6720 cpu_manager.go:414] "containerMap: removing container" podUID="13913563-7222-407c-98b1-5c1a17d10b82" containerName="istio-proxy" Oct 30 22:24:21.037724 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:21.037657 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13913563-7222-407c-98b1-5c1a17d10b82" containerName="istio-proxy" Oct 30 22:24:21.037724 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:21.037669 6720 cpu_manager.go:414] "containerMap: removing container" podUID="ad5f36c9-9a30-43b1-834b-3ab9c458bdb0" containerName="istio-proxy" Oct 30 22:24:21.037724 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:21.037677 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad5f36c9-9a30-43b1-834b-3ab9c458bdb0" containerName="istio-proxy" Oct 30 22:24:21.037724 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:21.037686 6720 cpu_manager.go:414] "containerMap: removing container" podUID="63d87167-397f-476b-837f-e40c8f14a994" containerName="istio-proxy" Oct 30 22:24:21.037724 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:21.037694 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63d87167-397f-476b-837f-e40c8f14a994" containerName="istio-proxy" Oct 30 22:24:21.037724 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:21.037721 6720 cpu_manager.go:414] "containerMap: removing container" podUID="0bd82fe0-1de3-40c8-b825-51eff2ae3386" containerName="istio-proxy" Oct 30 22:24:21.038435 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:21.037730 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bd82fe0-1de3-40c8-b825-51eff2ae3386" containerName="istio-proxy" Oct 30 22:24:21.038435 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:21.037883 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bd82fe0-1de3-40c8-b825-51eff2ae3386" containerName="istio-proxy" Oct 30 22:24:21.038435 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:21.037900 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="de3025de-693a-4527-a04e-c9dd6d3caf89" containerName="istio-proxy" Oct 30 22:24:21.038435 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:21.037913 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="28b34a7e-4765-46d0-b412-74eb9a1567d9" containerName="istio-proxy" Oct 30 22:24:21.038435 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:21.037923 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="5359ba6b-7deb-4055-9069-1e59a9c4e392" containerName="istio-proxy" Oct 30 22:24:21.038435 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:21.037938 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="0051f207-fce7-4aed-bcc3-a5aa69bb9a3c" containerName="istio-proxy" Oct 30 22:24:21.038435 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:21.037950 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="13913563-7222-407c-98b1-5c1a17d10b82" containerName="istio-proxy" Oct 30 22:24:21.038435 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:21.037958 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad5f36c9-9a30-43b1-834b-3ab9c458bdb0" containerName="istio-proxy" Oct 30 22:24:21.038435 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:21.037967 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="63d87167-397f-476b-837f-e40c8f14a994" containerName="istio-proxy" Oct 30 22:24:21.041980 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:21.041959 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-7pjjd" Oct 30 22:24:21.047022 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:21.047002 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-28b4bb-openshift-default-dockercfg-bdj78\"" Oct 30 22:24:21.054274 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:21.054250 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-7pjjd"] Oct 30 22:24:21.183314 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:21.183273 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d31cf31e-58d7-4ace-8959-6cba1429daec-istio-envoy\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-7pjjd\" (UID: \"d31cf31e-58d7-4ace-8959-6cba1429daec\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-7pjjd" Oct 30 22:24:21.183314 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:21.183313 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d31cf31e-58d7-4ace-8959-6cba1429daec-credential-socket\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-7pjjd\" (UID: \"d31cf31e-58d7-4ace-8959-6cba1429daec\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-7pjjd" Oct 30 22:24:21.183562 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:21.183337 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72ptn\" (UniqueName: \"kubernetes.io/projected/d31cf31e-58d7-4ace-8959-6cba1429daec-kube-api-access-72ptn\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-7pjjd\" (UID: \"d31cf31e-58d7-4ace-8959-6cba1429daec\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-7pjjd" Oct 30 22:24:21.183602 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:21.183576 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d31cf31e-58d7-4ace-8959-6cba1429daec-workload-certs\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-7pjjd\" (UID: \"d31cf31e-58d7-4ace-8959-6cba1429daec\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-7pjjd" Oct 30 22:24:21.183643 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:21.183615 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d31cf31e-58d7-4ace-8959-6cba1429daec-istio-data\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-7pjjd\" (UID: \"d31cf31e-58d7-4ace-8959-6cba1429daec\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-7pjjd" Oct 30 22:24:21.183758 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:21.183742 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d31cf31e-58d7-4ace-8959-6cba1429daec-istio-token\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-7pjjd\" (UID: \"d31cf31e-58d7-4ace-8959-6cba1429daec\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-7pjjd" Oct 30 22:24:21.183807 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:21.183795 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d31cf31e-58d7-4ace-8959-6cba1429daec-workload-socket\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-7pjjd\" (UID: \"d31cf31e-58d7-4ace-8959-6cba1429daec\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-7pjjd" Oct 30 22:24:21.183866 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:21.183850 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d31cf31e-58d7-4ace-8959-6cba1429daec-istiod-ca-cert\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-7pjjd\" (UID: \"d31cf31e-58d7-4ace-8959-6cba1429daec\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-7pjjd" Oct 30 22:24:21.183913 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:21.183903 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d31cf31e-58d7-4ace-8959-6cba1429daec-istio-podinfo\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-7pjjd\" (UID: \"d31cf31e-58d7-4ace-8959-6cba1429daec\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-7pjjd" Oct 30 22:24:21.284977 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:21.284938 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d31cf31e-58d7-4ace-8959-6cba1429daec-istio-token\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-7pjjd\" (UID: \"d31cf31e-58d7-4ace-8959-6cba1429daec\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-7pjjd" Oct 30 22:24:21.285202 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:21.285051 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d31cf31e-58d7-4ace-8959-6cba1429daec-workload-socket\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-7pjjd\" (UID: \"d31cf31e-58d7-4ace-8959-6cba1429daec\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-7pjjd" Oct 30 22:24:21.285202 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:21.285078 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d31cf31e-58d7-4ace-8959-6cba1429daec-istiod-ca-cert\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-7pjjd\" (UID: \"d31cf31e-58d7-4ace-8959-6cba1429daec\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-7pjjd" Oct 30 22:24:21.285202 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:21.285112 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d31cf31e-58d7-4ace-8959-6cba1429daec-istio-podinfo\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-7pjjd\" (UID: \"d31cf31e-58d7-4ace-8959-6cba1429daec\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-7pjjd" Oct 30 22:24:21.285202 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:21.285194 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d31cf31e-58d7-4ace-8959-6cba1429daec-istio-envoy\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-7pjjd\" (UID: \"d31cf31e-58d7-4ace-8959-6cba1429daec\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-7pjjd" Oct 30 22:24:21.285468 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:21.285219 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d31cf31e-58d7-4ace-8959-6cba1429daec-credential-socket\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-7pjjd\" (UID: \"d31cf31e-58d7-4ace-8959-6cba1429daec\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-7pjjd" Oct 30 22:24:21.285468 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:21.285250 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72ptn\" (UniqueName: \"kubernetes.io/projected/d31cf31e-58d7-4ace-8959-6cba1429daec-kube-api-access-72ptn\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-7pjjd\" (UID: \"d31cf31e-58d7-4ace-8959-6cba1429daec\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-7pjjd" Oct 30 22:24:21.285468 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:21.285363 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d31cf31e-58d7-4ace-8959-6cba1429daec-workload-certs\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-7pjjd\" (UID: \"d31cf31e-58d7-4ace-8959-6cba1429daec\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-7pjjd" Oct 30 22:24:21.285468 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:21.285418 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d31cf31e-58d7-4ace-8959-6cba1429daec-istio-data\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-7pjjd\" (UID: \"d31cf31e-58d7-4ace-8959-6cba1429daec\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-7pjjd" Oct 30 22:24:21.285653 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:21.285604 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d31cf31e-58d7-4ace-8959-6cba1429daec-credential-socket\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-7pjjd\" (UID: \"d31cf31e-58d7-4ace-8959-6cba1429daec\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-7pjjd" Oct 30 22:24:21.285653 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:21.285624 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d31cf31e-58d7-4ace-8959-6cba1429daec-workload-socket\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-7pjjd\" (UID: \"d31cf31e-58d7-4ace-8959-6cba1429daec\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-7pjjd" Oct 30 22:24:21.285896 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:21.285863 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d31cf31e-58d7-4ace-8959-6cba1429daec-istio-data\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-7pjjd\" (UID: \"d31cf31e-58d7-4ace-8959-6cba1429daec\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-7pjjd" Oct 30 22:24:21.285968 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:21.285896 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d31cf31e-58d7-4ace-8959-6cba1429daec-workload-certs\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-7pjjd\" (UID: \"d31cf31e-58d7-4ace-8959-6cba1429daec\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-7pjjd" Oct 30 22:24:21.286248 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:21.286226 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d31cf31e-58d7-4ace-8959-6cba1429daec-istiod-ca-cert\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-7pjjd\" (UID: \"d31cf31e-58d7-4ace-8959-6cba1429daec\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-7pjjd" Oct 30 22:24:21.288025 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:21.287965 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d31cf31e-58d7-4ace-8959-6cba1429daec-istio-envoy\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-7pjjd\" (UID: \"d31cf31e-58d7-4ace-8959-6cba1429daec\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-7pjjd" Oct 30 22:24:21.288336 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:21.288314 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d31cf31e-58d7-4ace-8959-6cba1429daec-istio-podinfo\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-7pjjd\" (UID: \"d31cf31e-58d7-4ace-8959-6cba1429daec\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-7pjjd" Oct 30 22:24:21.297751 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:21.297727 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d31cf31e-58d7-4ace-8959-6cba1429daec-istio-token\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-7pjjd\" (UID: \"d31cf31e-58d7-4ace-8959-6cba1429daec\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-7pjjd" Oct 30 22:24:21.297848 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:21.297799 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72ptn\" (UniqueName: \"kubernetes.io/projected/d31cf31e-58d7-4ace-8959-6cba1429daec-kube-api-access-72ptn\") pod \"e2e-extmod-b-28b4bb-openshift-default-588879f6bf-7pjjd\" (UID: \"d31cf31e-58d7-4ace-8959-6cba1429daec\") " pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-7pjjd" Oct 30 22:24:21.373439 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:21.354214 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:21.373439 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:21.367436 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-7pjjd" Oct 30 22:24:21.676884 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:21.676862 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-7pjjd"] Oct 30 22:24:21.679277 ip-10-0-137-181 kubenswrapper[6720]: W1030 22:24:21.679241 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd31cf31e_58d7_4ace_8959_6cba1429daec.slice/crio-747f0719d0615579cbe69cbe39dd43f10e9cd98eeb98cd4f354b515fe2cedcc5 WatchSource:0}: Error finding container 747f0719d0615579cbe69cbe39dd43f10e9cd98eeb98cd4f354b515fe2cedcc5: Status 404 returned error can't find the container with id 747f0719d0615579cbe69cbe39dd43f10e9cd98eeb98cd4f354b515fe2cedcc5 Oct 30 22:24:21.682264 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:21.682228 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:24:21.682360 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:21.682302 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:24:21.682360 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:21.682334 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:24:21.832963 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:21.832914 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-7pjjd" event={"ID":"d31cf31e-58d7-4ace-8959-6cba1429daec","Type":"ContainerStarted","Data":"beb3f5fb7f2a4ad15bd674de0f8cd0926921c23ffd279c401cabbdd8c414999f"} Oct 30 22:24:21.833157 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:21.832993 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-7pjjd" event={"ID":"d31cf31e-58d7-4ace-8959-6cba1429daec","Type":"ContainerStarted","Data":"747f0719d0615579cbe69cbe39dd43f10e9cd98eeb98cd4f354b515fe2cedcc5"} Oct 30 22:24:21.861719 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:21.861654 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-7pjjd" podStartSLOduration=1.861638704 podStartE2EDuration="1.861638704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:21.858989126 +0000 UTC m=+2628.029954357" watchObservedRunningTime="2026-10-30 22:24:21.861638704 +0000 UTC m=+2628.032603886" Oct 30 22:24:22.130414 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:22.128546 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:22.368587 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:22.368551 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-7pjjd" Oct 30 22:24:22.370438 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:22.370410 6720 patch_prober.go:29] interesting pod/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-7pjjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Oct 30 22:24:22.370588 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:22.370461 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-7pjjd" podUID="d31cf31e-58d7-4ace-8959-6cba1429daec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Oct 30 22:24:22.933600 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:22.933566 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:23.369827 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:23.369791 6720 patch_prober.go:29] interesting pod/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-7pjjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Oct 30 22:24:23.370056 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:23.369858 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-7pjjd" podUID="d31cf31e-58d7-4ace-8959-6cba1429daec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Oct 30 22:24:23.593105 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:23.591298 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:24.369466 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:24.368258 6720 patch_prober.go:29] interesting pod/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-7pjjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Oct 30 22:24:24.369466 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:24.368323 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-7pjjd" podUID="d31cf31e-58d7-4ace-8959-6cba1429daec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Oct 30 22:24:24.439477 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:24.438734 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:25.138037 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:25.137990 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:25.369797 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:25.369760 6720 patch_prober.go:29] interesting pod/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-7pjjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Oct 30 22:24:25.370242 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:25.369812 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-7pjjd" podUID="d31cf31e-58d7-4ace-8959-6cba1429daec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Oct 30 22:24:25.862420 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:25.860843 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:26.370858 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:26.370822 6720 patch_prober.go:29] interesting pod/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-7pjjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Oct 30 22:24:26.371340 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:26.370904 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-7pjjd" podUID="d31cf31e-58d7-4ace-8959-6cba1429daec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Oct 30 22:24:26.768093 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:26.767979 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:27.368885 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:27.368844 6720 patch_prober.go:29] interesting pod/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-7pjjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Oct 30 22:24:27.371712 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:27.369106 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-7pjjd" podUID="d31cf31e-58d7-4ace-8959-6cba1429daec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Oct 30 22:24:27.645471 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:27.640450 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:28.335508 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:28.333585 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:28.368344 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:28.368312 6720 patch_prober.go:29] interesting pod/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-7pjjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Oct 30 22:24:28.368547 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:28.368379 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-7pjjd" podUID="d31cf31e-58d7-4ace-8959-6cba1429daec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Oct 30 22:24:28.950904 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:28.950866 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:29.374159 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:29.374129 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-7pjjd" Oct 30 22:24:29.374464 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:29.374440 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-7pjjd" Oct 30 22:24:29.375450 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:29.375431 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-7pjjd" Oct 30 22:24:29.661239 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:29.661105 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:29.813934 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:29.813898 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:30.428714 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:30.428676 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:31.117375 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:31.117340 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:32.178910 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:32.178880 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:32.339036 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:32.338996 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:33.073439 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:33.071725 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:33.890408 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:33.890281 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:34.115074 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:34.115033 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:34.269145 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:34.269057 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:34.517456 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:34.505665 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:34.666997 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:34.663865 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:35.025404 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:35.025308 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:35.164659 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:35.164629 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:35.422830 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:35.422784 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:35.634971 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:35.634941 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:36.342094 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:36.342055 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:37.030117 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:37.030083 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:37.725752 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:37.725718 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:38.349416 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:38.349361 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:39.047562 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:39.043492 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:39.773410 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:39.761861 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:40.470825 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:24:40.470780 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 22:24:40.473278 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:40.473238 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:41.131203 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:41.131162 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:41.779550 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:41.779516 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:41.955170 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:41.955134 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:42.247947 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:42.247914 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:42.698780 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:42.698739 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:42.858692 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:42.858626 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:43.327155 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:43.327122 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:43.477708 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:43.477677 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:43.648901 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:43.648848 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:43.805664 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:43.805629 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:44.314213 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:44.314177 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:44.467507 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:44.467421 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:45.107102 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:45.106959 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:46.181138 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:46.181103 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:46.694111 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:46.694077 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:46.842271 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:46.842231 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:47.175938 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:47.175819 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:47.545112 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:47.544992 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:47.699754 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:47.699715 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:47.948372 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:47.948336 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:48.141413 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:48.129868 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:48.311878 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:48.311792 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:48.902770 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:48.902734 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:49.177503 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:49.177375 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:49.909126 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:49.909075 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:50.720868 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:50.720784 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:51.400410 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:51.400289 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:52.316411 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:52.314155 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:53.287753 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:53.287705 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:53.931902 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:53.931822 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:54.542723 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:54.542625 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:55.154457 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:55.154411 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:55.835286 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:55.834104 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:56.525415 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:56.522246 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:57.204029 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:57.203960 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:57.925991 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:57.925952 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:58.528375 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:58.528335 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:59.237841 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:59.237803 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:24:59.848971 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:24:59.848935 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:00.540787 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:00.540737 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:00.681320 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:00.681288 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:01.145581 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:01.145540 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:01.492793 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:01.492709 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:01.791126 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:01.791030 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:01.944413 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:01.942197 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:02.644406 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:02.644334 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:02.896505 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:02.896426 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:03.376762 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:03.376720 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:03.528777 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:03.528729 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:03.678259 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:03.678176 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:04.288264 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:04.288223 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:04.432418 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:04.432358 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:04.567620 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:04.567533 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:04.942407 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:04.942268 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:05.467634 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:05.467597 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:05.716412 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:05.714121 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:05.925226 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:05.925135 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:06.332209 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:06.332110 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-120cb1-6897f8fbbb-jszpt"] Oct 30 22:25:06.349020 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:06.348165 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-120cb1-6897f8fbbb-jszpt"] Oct 30 22:25:06.349020 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:06.348261 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-120cb1-6897f8fbbb-jszpt" Oct 30 22:25:06.354912 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:06.354596 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-120cb1-serving-cert\"" Oct 30 22:25:06.392090 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:06.392027 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:06.430708 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:06.420613 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24a24a49-0696-4e41-8b93-0b11814f98cd-metrics-certs\") pod \"maas-api-e2e-inf-a-120cb1-6897f8fbbb-jszpt\" (UID: \"24a24a49-0696-4e41-8b93-0b11814f98cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-120cb1-6897f8fbbb-jszpt" Oct 30 22:25:06.430708 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:06.420850 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24a24a49-0696-4e41-8b93-0b11814f98cd-openshift-service-ca\") pod \"maas-api-e2e-inf-a-120cb1-6897f8fbbb-jszpt\" (UID: \"24a24a49-0696-4e41-8b93-0b11814f98cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-120cb1-6897f8fbbb-jszpt" Oct 30 22:25:06.430708 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:06.421018 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24a24a49-0696-4e41-8b93-0b11814f98cd-maas-api-tls\") pod \"maas-api-e2e-inf-a-120cb1-6897f8fbbb-jszpt\" (UID: \"24a24a49-0696-4e41-8b93-0b11814f98cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-120cb1-6897f8fbbb-jszpt" Oct 30 22:25:06.430708 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:06.421202 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2l47m\" (UniqueName: \"kubernetes.io/projected/24a24a49-0696-4e41-8b93-0b11814f98cd-kube-api-access-2l47m\") pod \"maas-api-e2e-inf-a-120cb1-6897f8fbbb-jszpt\" (UID: \"24a24a49-0696-4e41-8b93-0b11814f98cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-120cb1-6897f8fbbb-jszpt" Oct 30 22:25:06.522644 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:06.522607 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24a24a49-0696-4e41-8b93-0b11814f98cd-openshift-service-ca\") pod \"maas-api-e2e-inf-a-120cb1-6897f8fbbb-jszpt\" (UID: \"24a24a49-0696-4e41-8b93-0b11814f98cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-120cb1-6897f8fbbb-jszpt" Oct 30 22:25:06.522864 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:06.522778 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24a24a49-0696-4e41-8b93-0b11814f98cd-maas-api-tls\") pod \"maas-api-e2e-inf-a-120cb1-6897f8fbbb-jszpt\" (UID: \"24a24a49-0696-4e41-8b93-0b11814f98cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-120cb1-6897f8fbbb-jszpt" Oct 30 22:25:06.522947 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:06.522882 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l47m\" (UniqueName: \"kubernetes.io/projected/24a24a49-0696-4e41-8b93-0b11814f98cd-kube-api-access-2l47m\") pod \"maas-api-e2e-inf-a-120cb1-6897f8fbbb-jszpt\" (UID: \"24a24a49-0696-4e41-8b93-0b11814f98cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-120cb1-6897f8fbbb-jszpt" Oct 30 22:25:06.522947 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:25:06.522918 6720 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-120cb1-serving-cert: secret "maas-api-e2e-inf-a-120cb1-serving-cert" not found Oct 30 22:25:06.523071 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:25:06.523016 6720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24a24a49-0696-4e41-8b93-0b11814f98cd-maas-api-tls podName:24a24a49-0696-4e41-8b93-0b11814f98cd nodeName:}" failed. No retries permitted until 2026-10-30 22:25:07.022988587 +0000 UTC m=+2673.193953814 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/24a24a49-0696-4e41-8b93-0b11814f98cd-maas-api-tls") pod "maas-api-e2e-inf-a-120cb1-6897f8fbbb-jszpt" (UID: "24a24a49-0696-4e41-8b93-0b11814f98cd") : secret "maas-api-e2e-inf-a-120cb1-serving-cert" not found Oct 30 22:25:06.523136 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:06.523076 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24a24a49-0696-4e41-8b93-0b11814f98cd-metrics-certs\") pod \"maas-api-e2e-inf-a-120cb1-6897f8fbbb-jszpt\" (UID: \"24a24a49-0696-4e41-8b93-0b11814f98cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-120cb1-6897f8fbbb-jszpt" Oct 30 22:25:06.523321 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:06.523281 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24a24a49-0696-4e41-8b93-0b11814f98cd-openshift-service-ca\") pod \"maas-api-e2e-inf-a-120cb1-6897f8fbbb-jszpt\" (UID: \"24a24a49-0696-4e41-8b93-0b11814f98cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-120cb1-6897f8fbbb-jszpt" Oct 30 22:25:06.526113 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:06.526089 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24a24a49-0696-4e41-8b93-0b11814f98cd-metrics-certs\") pod \"maas-api-e2e-inf-a-120cb1-6897f8fbbb-jszpt\" (UID: \"24a24a49-0696-4e41-8b93-0b11814f98cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-120cb1-6897f8fbbb-jszpt" Oct 30 22:25:06.534988 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:06.534963 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:06.536735 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:06.536712 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l47m\" (UniqueName: \"kubernetes.io/projected/24a24a49-0696-4e41-8b93-0b11814f98cd-kube-api-access-2l47m\") pod \"maas-api-e2e-inf-a-120cb1-6897f8fbbb-jszpt\" (UID: \"24a24a49-0696-4e41-8b93-0b11814f98cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-120cb1-6897f8fbbb-jszpt" Oct 30 22:25:06.883044 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:06.882949 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:07.030364 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:07.030327 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24a24a49-0696-4e41-8b93-0b11814f98cd-maas-api-tls\") pod \"maas-api-e2e-inf-a-120cb1-6897f8fbbb-jszpt\" (UID: \"24a24a49-0696-4e41-8b93-0b11814f98cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-120cb1-6897f8fbbb-jszpt" Oct 30 22:25:07.033234 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:07.033205 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24a24a49-0696-4e41-8b93-0b11814f98cd-maas-api-tls\") pod \"maas-api-e2e-inf-a-120cb1-6897f8fbbb-jszpt\" (UID: \"24a24a49-0696-4e41-8b93-0b11814f98cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-120cb1-6897f8fbbb-jszpt" Oct 30 22:25:07.067098 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:07.067066 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:07.290463 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:07.290032 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-120cb1-6897f8fbbb-jszpt" Oct 30 22:25:07.607993 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:07.607958 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:07.723697 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:07.723675 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-120cb1-6897f8fbbb-jszpt"] Oct 30 22:25:07.725414 ip-10-0-137-181 kubenswrapper[6720]: W1030 22:25:07.725342 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24a24a49_0696_4e41_8b93_0b11814f98cd.slice/crio-d3e27568b8a3633ce0f2b30ce325a34381d0f49a52093c43d9266d3b4b67cbc2 WatchSource:0}: Error finding container d3e27568b8a3633ce0f2b30ce325a34381d0f49a52093c43d9266d3b4b67cbc2: Status 404 returned error can't find the container with id d3e27568b8a3633ce0f2b30ce325a34381d0f49a52093c43d9266d3b4b67cbc2 Oct 30 22:25:07.726939 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:07.726921 6720 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:25:07.759961 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:07.759929 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:08.029462 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:08.028240 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:08.359675 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:08.359648 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-120cb1-6897f8fbbb-jszpt" event={"ID":"24a24a49-0696-4e41-8b93-0b11814f98cd","Type":"ContainerStarted","Data":"d3e27568b8a3633ce0f2b30ce325a34381d0f49a52093c43d9266d3b4b67cbc2"} Oct 30 22:25:08.846982 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:08.846942 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:09.065453 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:09.063125 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:09.219200 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:09.219155 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-df200a-7684bb76b6-rjq5j"] Oct 30 22:25:09.225104 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:09.225077 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df200a-7684bb76b6-rjq5j" Oct 30 22:25:09.232173 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:09.231923 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-df200a-serving-cert\"" Oct 30 22:25:09.239422 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:09.239263 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-df200a-7684bb76b6-rjq5j"] Oct 30 22:25:09.300829 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:09.300795 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f53144d-ae7f-493c-ac5b-714b9ad3b29e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-df200a-7684bb76b6-rjq5j\" (UID: \"1f53144d-ae7f-493c-ac5b-714b9ad3b29e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df200a-7684bb76b6-rjq5j" Oct 30 22:25:09.301075 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:09.301050 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrnqw\" (UniqueName: \"kubernetes.io/projected/1f53144d-ae7f-493c-ac5b-714b9ad3b29e-kube-api-access-rrnqw\") pod \"maas-api-e2e-inf-a-df200a-7684bb76b6-rjq5j\" (UID: \"1f53144d-ae7f-493c-ac5b-714b9ad3b29e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df200a-7684bb76b6-rjq5j" Oct 30 22:25:09.301178 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:09.301162 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f53144d-ae7f-493c-ac5b-714b9ad3b29e-maas-api-tls\") pod \"maas-api-e2e-inf-a-df200a-7684bb76b6-rjq5j\" (UID: \"1f53144d-ae7f-493c-ac5b-714b9ad3b29e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df200a-7684bb76b6-rjq5j" Oct 30 22:25:09.301280 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:09.301265 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f53144d-ae7f-493c-ac5b-714b9ad3b29e-metrics-certs\") pod \"maas-api-e2e-inf-a-df200a-7684bb76b6-rjq5j\" (UID: \"1f53144d-ae7f-493c-ac5b-714b9ad3b29e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df200a-7684bb76b6-rjq5j" Oct 30 22:25:09.340046 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:09.340009 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:09.368641 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:09.368534 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-120cb1-6897f8fbbb-jszpt" event={"ID":"24a24a49-0696-4e41-8b93-0b11814f98cd","Type":"ContainerStarted","Data":"591a61d66cf60d06f073464299527bc4fab4938da2a831c4c2198c4417400fc1"} Oct 30 22:25:09.368641 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:09.368604 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-120cb1-6897f8fbbb-jszpt" Oct 30 22:25:09.410113 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:09.410074 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrnqw\" (UniqueName: \"kubernetes.io/projected/1f53144d-ae7f-493c-ac5b-714b9ad3b29e-kube-api-access-rrnqw\") pod \"maas-api-e2e-inf-a-df200a-7684bb76b6-rjq5j\" (UID: \"1f53144d-ae7f-493c-ac5b-714b9ad3b29e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df200a-7684bb76b6-rjq5j" Oct 30 22:25:09.410316 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:09.410202 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f53144d-ae7f-493c-ac5b-714b9ad3b29e-maas-api-tls\") pod \"maas-api-e2e-inf-a-df200a-7684bb76b6-rjq5j\" (UID: \"1f53144d-ae7f-493c-ac5b-714b9ad3b29e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df200a-7684bb76b6-rjq5j" Oct 30 22:25:09.410469 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:09.410408 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f53144d-ae7f-493c-ac5b-714b9ad3b29e-metrics-certs\") pod \"maas-api-e2e-inf-a-df200a-7684bb76b6-rjq5j\" (UID: \"1f53144d-ae7f-493c-ac5b-714b9ad3b29e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df200a-7684bb76b6-rjq5j" Oct 30 22:25:09.410541 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:09.410485 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f53144d-ae7f-493c-ac5b-714b9ad3b29e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-df200a-7684bb76b6-rjq5j\" (UID: \"1f53144d-ae7f-493c-ac5b-714b9ad3b29e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df200a-7684bb76b6-rjq5j" Oct 30 22:25:09.412335 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:09.411372 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f53144d-ae7f-493c-ac5b-714b9ad3b29e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-df200a-7684bb76b6-rjq5j\" (UID: \"1f53144d-ae7f-493c-ac5b-714b9ad3b29e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df200a-7684bb76b6-rjq5j" Oct 30 22:25:09.432657 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:09.430756 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f53144d-ae7f-493c-ac5b-714b9ad3b29e-maas-api-tls\") pod \"maas-api-e2e-inf-a-df200a-7684bb76b6-rjq5j\" (UID: \"1f53144d-ae7f-493c-ac5b-714b9ad3b29e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df200a-7684bb76b6-rjq5j" Oct 30 22:25:09.442161 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:09.441823 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrnqw\" (UniqueName: \"kubernetes.io/projected/1f53144d-ae7f-493c-ac5b-714b9ad3b29e-kube-api-access-rrnqw\") pod \"maas-api-e2e-inf-a-df200a-7684bb76b6-rjq5j\" (UID: \"1f53144d-ae7f-493c-ac5b-714b9ad3b29e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df200a-7684bb76b6-rjq5j" Oct 30 22:25:09.445950 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:09.443442 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-120cb1-6897f8fbbb-jszpt" podStartSLOduration=2.834266209 podStartE2EDuration="3.44342715s" totalImagesPullingTime="609.160941ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:25:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:25:09.430840222 +0000 UTC m=+2675.601805438" watchObservedRunningTime="2026-10-30 22:25:09.44342715 +0000 UTC m=+2675.614392379" Oct 30 22:25:09.445950 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:09.444072 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f53144d-ae7f-493c-ac5b-714b9ad3b29e-metrics-certs\") pod \"maas-api-e2e-inf-a-df200a-7684bb76b6-rjq5j\" (UID: \"1f53144d-ae7f-493c-ac5b-714b9ad3b29e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df200a-7684bb76b6-rjq5j" Oct 30 22:25:09.485462 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:09.485370 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:09.562435 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:09.559951 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df200a-7684bb76b6-rjq5j" Oct 30 22:25:09.624434 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:09.624401 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:09.939354 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:09.939311 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-df200a-7684bb76b6-rjq5j"] Oct 30 22:25:10.316086 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:10.315994 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:10.380109 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:10.380067 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df200a-7684bb76b6-rjq5j" event={"ID":"1f53144d-ae7f-493c-ac5b-714b9ad3b29e","Type":"ContainerStarted","Data":"95fe21c7b7404ab25dc9017d3af9d3667ad290a8eac1c0acaf6d6d15af6aee75"} Oct 30 22:25:10.408594 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:25:10.408559 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 22:25:11.150409 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:11.147120 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:11.388196 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:11.388159 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df200a-7684bb76b6-rjq5j" event={"ID":"1f53144d-ae7f-493c-ac5b-714b9ad3b29e","Type":"ContainerStarted","Data":"9f0e80e6c01b1807d237a38be40889f98212dc1320b69e02c6884215a5643fc5"} Oct 30 22:25:11.388414 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:11.388257 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df200a-7684bb76b6-rjq5j" Oct 30 22:25:11.411149 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:11.411060 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df200a-7684bb76b6-rjq5j" podStartSLOduration=1.9089869670000001 podStartE2EDuration="2.411046465s" totalImagesPullingTime="502.059498ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:25:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:25:11.409589195 +0000 UTC m=+2677.580554380" watchObservedRunningTime="2026-10-30 22:25:11.411046465 +0000 UTC m=+2677.582012041" Oct 30 22:25:11.858410 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:11.853923 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:12.642426 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:12.639905 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:13.326411 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:13.324166 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:13.930165 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:13.930054 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:14.633971 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:14.633902 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:15.389727 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:15.389696 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-120cb1-6897f8fbbb-jszpt" Oct 30 22:25:15.523733 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:15.523695 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:16.427136 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:16.427089 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:17.442110 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:17.438744 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df200a-7684bb76b6-rjq5j" Oct 30 22:25:17.874413 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:17.870349 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:18.444157 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:18.443010 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:18.611648 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:18.611617 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:18.757703 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:18.757583 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:19.076545 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:19.076451 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:19.906919 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:19.906872 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:20.636945 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:20.636910 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:21.327414 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:21.326201 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:21.436818 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:21.436778 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k5rml"] Oct 30 22:25:21.437022 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:21.436999 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k5rml" podUID="2fa558cd-48e6-425c-a2cc-f4a75d24a20f" containerName="discovery" containerID="cri-o://09a94182d9eea04ca8720c9c623359521cd3f0cc2862820568dd33747bbf77c5" gracePeriod=30 Oct 30 22:25:21.737918 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:21.737879 6720 generic.go:356] "Generic (PLEG): container finished" podID="2fa558cd-48e6-425c-a2cc-f4a75d24a20f" containerID="09a94182d9eea04ca8720c9c623359521cd3f0cc2862820568dd33747bbf77c5" exitCode=0 Oct 30 22:25:21.738343 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:21.738157 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k5rml" event={"ID":"2fa558cd-48e6-425c-a2cc-f4a75d24a20f","Type":"ContainerDied","Data":"09a94182d9eea04ca8720c9c623359521cd3f0cc2862820568dd33747bbf77c5"} Oct 30 22:25:21.788198 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:21.788169 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k5rml" Oct 30 22:25:21.906726 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:21.906637 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fa558cd-48e6-425c-a2cc-f4a75d24a20f-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2fa558cd-48e6-425c-a2cc-f4a75d24a20f-istio-kubeconfig\") pod \"2fa558cd-48e6-425c-a2cc-f4a75d24a20f\" (UID: \"2fa558cd-48e6-425c-a2cc-f4a75d24a20f\") " Oct 30 22:25:21.906871 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:21.906726 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fa558cd-48e6-425c-a2cc-f4a75d24a20f-kube-api-access-9bscf\" (UniqueName: \"kubernetes.io/projected/2fa558cd-48e6-425c-a2cc-f4a75d24a20f-kube-api-access-9bscf\") pod \"2fa558cd-48e6-425c-a2cc-f4a75d24a20f\" (UID: \"2fa558cd-48e6-425c-a2cc-f4a75d24a20f\") " Oct 30 22:25:21.906871 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:21.906788 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fa558cd-48e6-425c-a2cc-f4a75d24a20f-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2fa558cd-48e6-425c-a2cc-f4a75d24a20f-istio-csr-ca-configmap\") pod \"2fa558cd-48e6-425c-a2cc-f4a75d24a20f\" (UID: \"2fa558cd-48e6-425c-a2cc-f4a75d24a20f\") " Oct 30 22:25:21.906871 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:21.906829 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fa558cd-48e6-425c-a2cc-f4a75d24a20f-istio-token\" (UniqueName: \"kubernetes.io/projected/2fa558cd-48e6-425c-a2cc-f4a75d24a20f-istio-token\") pod \"2fa558cd-48e6-425c-a2cc-f4a75d24a20f\" (UID: \"2fa558cd-48e6-425c-a2cc-f4a75d24a20f\") " Oct 30 22:25:21.906871 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:21.906850 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fa558cd-48e6-425c-a2cc-f4a75d24a20f-cacerts\" (UniqueName: \"kubernetes.io/secret/2fa558cd-48e6-425c-a2cc-f4a75d24a20f-cacerts\") pod \"2fa558cd-48e6-425c-a2cc-f4a75d24a20f\" (UID: \"2fa558cd-48e6-425c-a2cc-f4a75d24a20f\") " Oct 30 22:25:21.907091 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:21.906951 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fa558cd-48e6-425c-a2cc-f4a75d24a20f-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2fa558cd-48e6-425c-a2cc-f4a75d24a20f-istio-csr-dns-cert\") pod \"2fa558cd-48e6-425c-a2cc-f4a75d24a20f\" (UID: \"2fa558cd-48e6-425c-a2cc-f4a75d24a20f\") " Oct 30 22:25:21.907136 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:21.907091 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2fa558cd-48e6-425c-a2cc-f4a75d24a20f-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fa558cd-48e6-425c-a2cc-f4a75d24a20f-local-certs\") pod \"2fa558cd-48e6-425c-a2cc-f4a75d24a20f\" (UID: \"2fa558cd-48e6-425c-a2cc-f4a75d24a20f\") " Oct 30 22:25:21.907188 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:21.907144 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fa558cd-48e6-425c-a2cc-f4a75d24a20f-istio-csr-ca-configmap" pod "2fa558cd-48e6-425c-a2cc-f4a75d24a20f" (UID: "2fa558cd-48e6-425c-a2cc-f4a75d24a20f"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:25:21.908125 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:21.908084 6720 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2fa558cd-48e6-425c-a2cc-f4a75d24a20f-istio-csr-ca-configmap\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:21.909676 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:21.909650 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fa558cd-48e6-425c-a2cc-f4a75d24a20f-cacerts" pod "2fa558cd-48e6-425c-a2cc-f4a75d24a20f" (UID: "2fa558cd-48e6-425c-a2cc-f4a75d24a20f"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:25:21.909888 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:21.909859 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fa558cd-48e6-425c-a2cc-f4a75d24a20f-kube-api-access-9bscf" pod "2fa558cd-48e6-425c-a2cc-f4a75d24a20f" (UID: "2fa558cd-48e6-425c-a2cc-f4a75d24a20f"). InnerVolumeSpecName "kube-api-access-9bscf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:21.909888 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:21.909880 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fa558cd-48e6-425c-a2cc-f4a75d24a20f-local-certs" pod "2fa558cd-48e6-425c-a2cc-f4a75d24a20f" (UID: "2fa558cd-48e6-425c-a2cc-f4a75d24a20f"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:21.910015 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:21.909894 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fa558cd-48e6-425c-a2cc-f4a75d24a20f-istio-csr-dns-cert" pod "2fa558cd-48e6-425c-a2cc-f4a75d24a20f" (UID: "2fa558cd-48e6-425c-a2cc-f4a75d24a20f"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:25:21.910130 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:21.910111 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fa558cd-48e6-425c-a2cc-f4a75d24a20f-istio-token" pod "2fa558cd-48e6-425c-a2cc-f4a75d24a20f" (UID: "2fa558cd-48e6-425c-a2cc-f4a75d24a20f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:21.910249 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:21.910227 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fa558cd-48e6-425c-a2cc-f4a75d24a20f-istio-kubeconfig" pod "2fa558cd-48e6-425c-a2cc-f4a75d24a20f" (UID: "2fa558cd-48e6-425c-a2cc-f4a75d24a20f"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:25:22.009873 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:22.009840 6720 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2fa558cd-48e6-425c-a2cc-f4a75d24a20f-istio-token\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:22.009873 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:22.009873 6720 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2fa558cd-48e6-425c-a2cc-f4a75d24a20f-cacerts\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:22.009873 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:22.009889 6720 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2fa558cd-48e6-425c-a2cc-f4a75d24a20f-istio-csr-dns-cert\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:22.010146 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:22.009906 6720 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2fa558cd-48e6-425c-a2cc-f4a75d24a20f-local-certs\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:22.010146 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:22.009986 6720 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2fa558cd-48e6-425c-a2cc-f4a75d24a20f-istio-kubeconfig\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:22.010146 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:22.010012 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bscf\" (UniqueName: \"kubernetes.io/projected/2fa558cd-48e6-425c-a2cc-f4a75d24a20f-kube-api-access-9bscf\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:22.150410 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:22.145812 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:22.781405 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:22.781291 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k5rml" event={"ID":"2fa558cd-48e6-425c-a2cc-f4a75d24a20f","Type":"ContainerDied","Data":"52a29f2340f696ef40512a7099c6bc61f1845592711b98331f82a4fa14877f46"} Oct 30 22:25:22.781948 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:22.781413 6720 scope.go:122] "RemoveContainer" containerID="09a94182d9eea04ca8720c9c623359521cd3f0cc2862820568dd33747bbf77c5" Oct 30 22:25:22.781948 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:22.781674 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k5rml" Oct 30 22:25:22.829527 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:22.829496 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k5rml"] Oct 30 22:25:22.834234 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:22.834184 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k5rml"] Oct 30 22:25:22.864418 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:22.852890 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:23.613425 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:23.612938 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:23.958483 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:23.958360 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fa558cd-48e6-425c-a2cc-f4a75d24a20f" path="/var/lib/kubelet/pods/2fa558cd-48e6-425c-a2cc-f4a75d24a20f/volumes" Oct 30 22:25:24.224902 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:24.224785 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:24.848869 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:24.843288 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:25.557180 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:25.557138 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:26.201045 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:26.200993 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:26.962192 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:26.962148 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:27.618016 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:27.617980 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-28d39b-5b4c7f5c97-qsnhq"] Oct 30 22:25:27.619068 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:27.619041 6720 cpu_manager.go:414] "containerMap: removing container" podUID="2fa558cd-48e6-425c-a2cc-f4a75d24a20f" containerName="discovery" Oct 30 22:25:27.619068 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:27.619066 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fa558cd-48e6-425c-a2cc-f4a75d24a20f" containerName="discovery" Oct 30 22:25:27.619292 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:27.619222 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fa558cd-48e6-425c-a2cc-f4a75d24a20f" containerName="discovery" Oct 30 22:25:27.625018 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:27.624959 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28d39b-5b4c7f5c97-qsnhq" Oct 30 22:25:27.631974 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:27.631948 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-28d39b-serving-cert\"" Oct 30 22:25:27.636018 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:27.635991 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-28d39b-5b4c7f5c97-qsnhq"] Oct 30 22:25:27.708555 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:27.708520 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:27.709029 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:27.709005 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a771bf56-fa60-4bac-bfd3-4aa0ccf2bf82-maas-api-tls\") pod \"maas-api-e2e-inf-a-28d39b-5b4c7f5c97-qsnhq\" (UID: \"a771bf56-fa60-4bac-bfd3-4aa0ccf2bf82\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28d39b-5b4c7f5c97-qsnhq" Oct 30 22:25:27.709234 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:27.709212 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a771bf56-fa60-4bac-bfd3-4aa0ccf2bf82-openshift-service-ca\") pod \"maas-api-e2e-inf-a-28d39b-5b4c7f5c97-qsnhq\" (UID: \"a771bf56-fa60-4bac-bfd3-4aa0ccf2bf82\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28d39b-5b4c7f5c97-qsnhq" Oct 30 22:25:27.709307 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:27.709291 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a771bf56-fa60-4bac-bfd3-4aa0ccf2bf82-metrics-certs\") pod \"maas-api-e2e-inf-a-28d39b-5b4c7f5c97-qsnhq\" (UID: \"a771bf56-fa60-4bac-bfd3-4aa0ccf2bf82\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28d39b-5b4c7f5c97-qsnhq" Oct 30 22:25:27.709364 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:27.709336 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4j4lm\" (UniqueName: \"kubernetes.io/projected/a771bf56-fa60-4bac-bfd3-4aa0ccf2bf82-kube-api-access-4j4lm\") pod \"maas-api-e2e-inf-a-28d39b-5b4c7f5c97-qsnhq\" (UID: \"a771bf56-fa60-4bac-bfd3-4aa0ccf2bf82\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28d39b-5b4c7f5c97-qsnhq" Oct 30 22:25:27.811755 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:27.811528 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a771bf56-fa60-4bac-bfd3-4aa0ccf2bf82-openshift-service-ca\") pod \"maas-api-e2e-inf-a-28d39b-5b4c7f5c97-qsnhq\" (UID: \"a771bf56-fa60-4bac-bfd3-4aa0ccf2bf82\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28d39b-5b4c7f5c97-qsnhq" Oct 30 22:25:27.812110 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:27.812083 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a771bf56-fa60-4bac-bfd3-4aa0ccf2bf82-metrics-certs\") pod \"maas-api-e2e-inf-a-28d39b-5b4c7f5c97-qsnhq\" (UID: \"a771bf56-fa60-4bac-bfd3-4aa0ccf2bf82\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28d39b-5b4c7f5c97-qsnhq" Oct 30 22:25:27.812308 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:27.812289 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j4lm\" (UniqueName: \"kubernetes.io/projected/a771bf56-fa60-4bac-bfd3-4aa0ccf2bf82-kube-api-access-4j4lm\") pod \"maas-api-e2e-inf-a-28d39b-5b4c7f5c97-qsnhq\" (UID: \"a771bf56-fa60-4bac-bfd3-4aa0ccf2bf82\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28d39b-5b4c7f5c97-qsnhq" Oct 30 22:25:27.812648 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:27.812628 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a771bf56-fa60-4bac-bfd3-4aa0ccf2bf82-maas-api-tls\") pod \"maas-api-e2e-inf-a-28d39b-5b4c7f5c97-qsnhq\" (UID: \"a771bf56-fa60-4bac-bfd3-4aa0ccf2bf82\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28d39b-5b4c7f5c97-qsnhq" Oct 30 22:25:27.817691 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:27.817663 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a771bf56-fa60-4bac-bfd3-4aa0ccf2bf82-openshift-service-ca\") pod \"maas-api-e2e-inf-a-28d39b-5b4c7f5c97-qsnhq\" (UID: \"a771bf56-fa60-4bac-bfd3-4aa0ccf2bf82\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28d39b-5b4c7f5c97-qsnhq" Oct 30 22:25:27.820835 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:27.820810 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a771bf56-fa60-4bac-bfd3-4aa0ccf2bf82-maas-api-tls\") pod \"maas-api-e2e-inf-a-28d39b-5b4c7f5c97-qsnhq\" (UID: \"a771bf56-fa60-4bac-bfd3-4aa0ccf2bf82\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28d39b-5b4c7f5c97-qsnhq" Oct 30 22:25:27.821303 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:27.821277 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a771bf56-fa60-4bac-bfd3-4aa0ccf2bf82-metrics-certs\") pod \"maas-api-e2e-inf-a-28d39b-5b4c7f5c97-qsnhq\" (UID: \"a771bf56-fa60-4bac-bfd3-4aa0ccf2bf82\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28d39b-5b4c7f5c97-qsnhq" Oct 30 22:25:27.829089 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:27.829066 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j4lm\" (UniqueName: \"kubernetes.io/projected/a771bf56-fa60-4bac-bfd3-4aa0ccf2bf82-kube-api-access-4j4lm\") pod \"maas-api-e2e-inf-a-28d39b-5b4c7f5c97-qsnhq\" (UID: \"a771bf56-fa60-4bac-bfd3-4aa0ccf2bf82\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28d39b-5b4c7f5c97-qsnhq" Oct 30 22:25:27.848852 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:27.848824 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:27.955609 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:27.955528 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28d39b-5b4c7f5c97-qsnhq" Oct 30 22:25:28.285002 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:28.284969 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-28d39b-5b4c7f5c97-qsnhq"] Oct 30 22:25:28.287162 ip-10-0-137-181 kubenswrapper[6720]: W1030 22:25:28.287119 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda771bf56_fa60_4bac_bfd3_4aa0ccf2bf82.slice/crio-162a7653a34d5148b8e1c19212a1672db18247424e714dac5565ff7835df8665 WatchSource:0}: Error finding container 162a7653a34d5148b8e1c19212a1672db18247424e714dac5565ff7835df8665: Status 404 returned error can't find the container with id 162a7653a34d5148b8e1c19212a1672db18247424e714dac5565ff7835df8665 Oct 30 22:25:28.586462 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:28.584165 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:28.831851 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:28.831813 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28d39b-5b4c7f5c97-qsnhq" event={"ID":"a771bf56-fa60-4bac-bfd3-4aa0ccf2bf82","Type":"ContainerStarted","Data":"cff26aa9634b9da60583f367d5540bcaae964a2b369db37edf93176fd2877780"} Oct 30 22:25:28.832019 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:28.831878 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28d39b-5b4c7f5c97-qsnhq" event={"ID":"a771bf56-fa60-4bac-bfd3-4aa0ccf2bf82","Type":"ContainerStarted","Data":"162a7653a34d5148b8e1c19212a1672db18247424e714dac5565ff7835df8665"} Oct 30 22:25:28.832019 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:28.831903 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28d39b-5b4c7f5c97-qsnhq" Oct 30 22:25:28.852581 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:28.852478 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28d39b-5b4c7f5c97-qsnhq" podStartSLOduration=1.401372797 podStartE2EDuration="1.852462867s" totalImagesPullingTime="451.09007ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:25:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:25:28.851749215 +0000 UTC m=+2695.022714408" watchObservedRunningTime="2026-10-30 22:25:28.852462867 +0000 UTC m=+2695.023428049" Oct 30 22:25:29.418295 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:29.418162 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:29.699159 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:29.699079 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:30.212445 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:30.212115 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:31.005887 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:31.004300 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:31.927935 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:31.927900 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:32.416245 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:32.416201 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:32.763514 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:32.763429 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:32.941111 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:32.941076 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:33.400334 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:33.400296 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:33.589724 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:33.589691 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:33.753032 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:33.752941 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:33.902804 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:33.902769 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:34.249430 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:34.248341 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:34.510940 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:34.510847 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:34.721416 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:34.714946 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:34.845204 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:34.845116 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28d39b-5b4c7f5c97-qsnhq" Oct 30 22:25:35.462425 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:35.462362 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:36.302671 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:36.302611 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:37.072947 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:37.072917 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:37.382199 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:37.382164 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:37.579901 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:37.579862 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:38.145291 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:38.145251 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:38.316713 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:38.316645 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:38.455362 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:38.455277 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:38.727284 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:38.727202 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:38.911559 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:38.911523 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:39.269037 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:39.269007 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:39.876858 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:39.876819 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:40.085229 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:40.085141 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:40.279373 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:40.279269 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:40.408174 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:25:40.408135 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 22:25:40.433043 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:40.433013 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:40.729558 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:40.729526 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:40.890355 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:40.890322 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:41.517646 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:41.517150 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:42.292452 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:42.291019 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:42.961109 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:42.959347 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:43.614499 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:43.613484 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:44.308075 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:44.308022 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:44.960948 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:44.960919 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:45.619875 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:45.619840 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:46.250123 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:46.250079 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:47.111282 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:47.111229 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:47.443129 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:47.443034 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:47.618573 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:47.618537 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:47.765634 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:47.765544 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:47.945315 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:47.945267 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:48.821190 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:48.821155 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:49.898312 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:49.898277 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:50.902416 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:50.901012 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:51.747722 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:51.747685 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:52.409197 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:52.409145 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:53.116509 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:53.116480 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:53.268575 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:53.268534 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:53.688407 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:53.686931 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:53.979598 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:53.979510 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:54.161366 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:54.161332 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:54.323878 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:54.323792 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:55.115613 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:55.115529 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:56.201884 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:56.200694 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:57.221598 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:57.221553 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:58.486962 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:58.486851 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:58.639253 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:58.639219 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:59.035379 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:59.035345 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:59.188255 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:59.188225 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:59.438871 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:59.438834 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4b2bb0-ffd47c566-qwcvz"] Oct 30 22:25:59.445574 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:59.445549 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b2bb0-ffd47c566-qwcvz" Oct 30 22:25:59.454045 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:59.454015 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-4b2bb0-serving-cert\"" Oct 30 22:25:59.456709 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:59.456683 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4b2bb0-ffd47c566-qwcvz"] Oct 30 22:25:59.563471 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:59.563361 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:59.614546 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:59.614514 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxq5v\" (UniqueName: \"kubernetes.io/projected/6b6fc7b5-8eca-4f2c-b012-6f9375211314-kube-api-access-sxq5v\") pod \"maas-api-e2e-inf-b-4b2bb0-ffd47c566-qwcvz\" (UID: \"6b6fc7b5-8eca-4f2c-b012-6f9375211314\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b2bb0-ffd47c566-qwcvz" Oct 30 22:25:59.614729 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:59.614572 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b6fc7b5-8eca-4f2c-b012-6f9375211314-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4b2bb0-ffd47c566-qwcvz\" (UID: \"6b6fc7b5-8eca-4f2c-b012-6f9375211314\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b2bb0-ffd47c566-qwcvz" Oct 30 22:25:59.614833 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:59.614812 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b6fc7b5-8eca-4f2c-b012-6f9375211314-metrics-certs\") pod \"maas-api-e2e-inf-b-4b2bb0-ffd47c566-qwcvz\" (UID: \"6b6fc7b5-8eca-4f2c-b012-6f9375211314\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b2bb0-ffd47c566-qwcvz" Oct 30 22:25:59.614871 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:59.614853 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b6fc7b5-8eca-4f2c-b012-6f9375211314-maas-api-tls\") pod \"maas-api-e2e-inf-b-4b2bb0-ffd47c566-qwcvz\" (UID: \"6b6fc7b5-8eca-4f2c-b012-6f9375211314\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b2bb0-ffd47c566-qwcvz" Oct 30 22:25:59.715913 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:59.715826 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b6fc7b5-8eca-4f2c-b012-6f9375211314-metrics-certs\") pod \"maas-api-e2e-inf-b-4b2bb0-ffd47c566-qwcvz\" (UID: \"6b6fc7b5-8eca-4f2c-b012-6f9375211314\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b2bb0-ffd47c566-qwcvz" Oct 30 22:25:59.715913 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:59.715872 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b6fc7b5-8eca-4f2c-b012-6f9375211314-maas-api-tls\") pod \"maas-api-e2e-inf-b-4b2bb0-ffd47c566-qwcvz\" (UID: \"6b6fc7b5-8eca-4f2c-b012-6f9375211314\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b2bb0-ffd47c566-qwcvz" Oct 30 22:25:59.716168 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:59.715925 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxq5v\" (UniqueName: \"kubernetes.io/projected/6b6fc7b5-8eca-4f2c-b012-6f9375211314-kube-api-access-sxq5v\") pod \"maas-api-e2e-inf-b-4b2bb0-ffd47c566-qwcvz\" (UID: \"6b6fc7b5-8eca-4f2c-b012-6f9375211314\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b2bb0-ffd47c566-qwcvz" Oct 30 22:25:59.716168 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:59.715980 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b6fc7b5-8eca-4f2c-b012-6f9375211314-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4b2bb0-ffd47c566-qwcvz\" (UID: \"6b6fc7b5-8eca-4f2c-b012-6f9375211314\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b2bb0-ffd47c566-qwcvz" Oct 30 22:25:59.716168 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:25:59.716011 6720 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-4b2bb0-serving-cert: secret "maas-api-e2e-inf-b-4b2bb0-serving-cert" not found Oct 30 22:25:59.716168 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:25:59.716092 6720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b6fc7b5-8eca-4f2c-b012-6f9375211314-maas-api-tls podName:6b6fc7b5-8eca-4f2c-b012-6f9375211314 nodeName:}" failed. No retries permitted until 2026-10-30 22:26:00.216068923 +0000 UTC m=+2726.387034097 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6b6fc7b5-8eca-4f2c-b012-6f9375211314-maas-api-tls") pod "maas-api-e2e-inf-b-4b2bb0-ffd47c566-qwcvz" (UID: "6b6fc7b5-8eca-4f2c-b012-6f9375211314") : secret "maas-api-e2e-inf-b-4b2bb0-serving-cert" not found Oct 30 22:25:59.716749 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:59.716728 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b6fc7b5-8eca-4f2c-b012-6f9375211314-openshift-service-ca\") pod \"maas-api-e2e-inf-b-4b2bb0-ffd47c566-qwcvz\" (UID: \"6b6fc7b5-8eca-4f2c-b012-6f9375211314\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b2bb0-ffd47c566-qwcvz" Oct 30 22:25:59.718757 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:59.718734 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b6fc7b5-8eca-4f2c-b012-6f9375211314-metrics-certs\") pod \"maas-api-e2e-inf-b-4b2bb0-ffd47c566-qwcvz\" (UID: \"6b6fc7b5-8eca-4f2c-b012-6f9375211314\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b2bb0-ffd47c566-qwcvz" Oct 30 22:25:59.730875 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:59.730849 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxq5v\" (UniqueName: \"kubernetes.io/projected/6b6fc7b5-8eca-4f2c-b012-6f9375211314-kube-api-access-sxq5v\") pod \"maas-api-e2e-inf-b-4b2bb0-ffd47c566-qwcvz\" (UID: \"6b6fc7b5-8eca-4f2c-b012-6f9375211314\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b2bb0-ffd47c566-qwcvz" Oct 30 22:25:59.766419 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:59.766358 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:25:59.988728 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:25:59.988613 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:00.244491 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:00.244263 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b6fc7b5-8eca-4f2c-b012-6f9375211314-maas-api-tls\") pod \"maas-api-e2e-inf-b-4b2bb0-ffd47c566-qwcvz\" (UID: \"6b6fc7b5-8eca-4f2c-b012-6f9375211314\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b2bb0-ffd47c566-qwcvz" Oct 30 22:26:00.248193 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:00.248163 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b6fc7b5-8eca-4f2c-b012-6f9375211314-maas-api-tls\") pod \"maas-api-e2e-inf-b-4b2bb0-ffd47c566-qwcvz\" (UID: \"6b6fc7b5-8eca-4f2c-b012-6f9375211314\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b2bb0-ffd47c566-qwcvz" Oct 30 22:26:00.374433 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:00.363035 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b2bb0-ffd47c566-qwcvz" Oct 30 22:26:00.474101 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:00.473962 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:00.724074 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:00.724038 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4b2bb0-ffd47c566-qwcvz"] Oct 30 22:26:00.762523 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:00.762490 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:00.968654 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:00.968556 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:01.152610 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:01.152575 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b2bb0-ffd47c566-qwcvz" event={"ID":"6b6fc7b5-8eca-4f2c-b012-6f9375211314","Type":"ContainerStarted","Data":"06178a2e194c31f8164770a03bbab663c8a0a8dd6b4ef78226861ebe57e4e00c"} Oct 30 22:26:01.819795 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:01.819757 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:02.161811 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:02.161778 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b2bb0-ffd47c566-qwcvz" event={"ID":"6b6fc7b5-8eca-4f2c-b012-6f9375211314","Type":"ContainerStarted","Data":"b670497784a390c55f50c438626e2e7c0e95b76477655655ee157d3527ec12fd"} Oct 30 22:26:02.162037 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:02.161860 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b2bb0-ffd47c566-qwcvz" Oct 30 22:26:02.189162 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:02.189105 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b2bb0-ffd47c566-qwcvz" podStartSLOduration=2.775072095 podStartE2EDuration="3.189091211s" totalImagesPullingTime="414.019116ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:25:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:02.187607493 +0000 UTC m=+2728.358572687" watchObservedRunningTime="2026-10-30 22:26:02.189091211 +0000 UTC m=+2728.360056392" Oct 30 22:26:02.632311 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:02.632275 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c92aec-5bd6c598b7-f86vh"] Oct 30 22:26:02.636794 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:02.636767 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c92aec-5bd6c598b7-f86vh" Oct 30 22:26:02.645328 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:02.645299 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-c92aec-serving-cert\"" Oct 30 22:26:02.650898 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:02.650866 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c92aec-5bd6c598b7-f86vh"] Oct 30 22:26:02.754963 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:02.753964 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:02.796601 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:02.796534 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22545d04-38b5-4d64-89a0-9e533875220e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c92aec-5bd6c598b7-f86vh\" (UID: \"22545d04-38b5-4d64-89a0-9e533875220e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c92aec-5bd6c598b7-f86vh" Oct 30 22:26:02.797656 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:02.797629 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62pgj\" (UniqueName: \"kubernetes.io/projected/22545d04-38b5-4d64-89a0-9e533875220e-kube-api-access-62pgj\") pod \"maas-api-e2e-inf-b-c92aec-5bd6c598b7-f86vh\" (UID: \"22545d04-38b5-4d64-89a0-9e533875220e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c92aec-5bd6c598b7-f86vh" Oct 30 22:26:02.798015 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:02.797993 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22545d04-38b5-4d64-89a0-9e533875220e-maas-api-tls\") pod \"maas-api-e2e-inf-b-c92aec-5bd6c598b7-f86vh\" (UID: \"22545d04-38b5-4d64-89a0-9e533875220e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c92aec-5bd6c598b7-f86vh" Oct 30 22:26:02.798170 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:02.798155 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22545d04-38b5-4d64-89a0-9e533875220e-metrics-certs\") pod \"maas-api-e2e-inf-b-c92aec-5bd6c598b7-f86vh\" (UID: \"22545d04-38b5-4d64-89a0-9e533875220e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c92aec-5bd6c598b7-f86vh" Oct 30 22:26:02.900052 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:02.899914 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22545d04-38b5-4d64-89a0-9e533875220e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c92aec-5bd6c598b7-f86vh\" (UID: \"22545d04-38b5-4d64-89a0-9e533875220e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c92aec-5bd6c598b7-f86vh" Oct 30 22:26:02.900599 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:02.900056 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62pgj\" (UniqueName: \"kubernetes.io/projected/22545d04-38b5-4d64-89a0-9e533875220e-kube-api-access-62pgj\") pod \"maas-api-e2e-inf-b-c92aec-5bd6c598b7-f86vh\" (UID: \"22545d04-38b5-4d64-89a0-9e533875220e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c92aec-5bd6c598b7-f86vh" Oct 30 22:26:02.900599 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:02.900110 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22545d04-38b5-4d64-89a0-9e533875220e-maas-api-tls\") pod \"maas-api-e2e-inf-b-c92aec-5bd6c598b7-f86vh\" (UID: \"22545d04-38b5-4d64-89a0-9e533875220e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c92aec-5bd6c598b7-f86vh" Oct 30 22:26:02.900599 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:02.900142 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22545d04-38b5-4d64-89a0-9e533875220e-metrics-certs\") pod \"maas-api-e2e-inf-b-c92aec-5bd6c598b7-f86vh\" (UID: \"22545d04-38b5-4d64-89a0-9e533875220e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c92aec-5bd6c598b7-f86vh" Oct 30 22:26:02.900776 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:02.900662 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22545d04-38b5-4d64-89a0-9e533875220e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c92aec-5bd6c598b7-f86vh\" (UID: \"22545d04-38b5-4d64-89a0-9e533875220e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c92aec-5bd6c598b7-f86vh" Oct 30 22:26:02.900776 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:26:02.900770 6720 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-c92aec-serving-cert: secret "maas-api-e2e-inf-b-c92aec-serving-cert" not found Oct 30 22:26:02.900882 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:26:02.900839 6720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22545d04-38b5-4d64-89a0-9e533875220e-maas-api-tls podName:22545d04-38b5-4d64-89a0-9e533875220e nodeName:}" failed. No retries permitted until 2026-10-30 22:26:03.400821054 +0000 UTC m=+2729.571786227 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/22545d04-38b5-4d64-89a0-9e533875220e-maas-api-tls") pod "maas-api-e2e-inf-b-c92aec-5bd6c598b7-f86vh" (UID: "22545d04-38b5-4d64-89a0-9e533875220e") : secret "maas-api-e2e-inf-b-c92aec-serving-cert" not found Oct 30 22:26:02.910475 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:02.903276 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22545d04-38b5-4d64-89a0-9e533875220e-metrics-certs\") pod \"maas-api-e2e-inf-b-c92aec-5bd6c598b7-f86vh\" (UID: \"22545d04-38b5-4d64-89a0-9e533875220e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c92aec-5bd6c598b7-f86vh" Oct 30 22:26:02.913412 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:02.913224 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:02.914656 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:02.914625 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62pgj\" (UniqueName: \"kubernetes.io/projected/22545d04-38b5-4d64-89a0-9e533875220e-kube-api-access-62pgj\") pod \"maas-api-e2e-inf-b-c92aec-5bd6c598b7-f86vh\" (UID: \"22545d04-38b5-4d64-89a0-9e533875220e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c92aec-5bd6c598b7-f86vh" Oct 30 22:26:03.077739 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:03.077704 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:03.414409 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:03.412619 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22545d04-38b5-4d64-89a0-9e533875220e-maas-api-tls\") pod \"maas-api-e2e-inf-b-c92aec-5bd6c598b7-f86vh\" (UID: \"22545d04-38b5-4d64-89a0-9e533875220e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c92aec-5bd6c598b7-f86vh" Oct 30 22:26:03.450294 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:03.450255 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22545d04-38b5-4d64-89a0-9e533875220e-maas-api-tls\") pod \"maas-api-e2e-inf-b-c92aec-5bd6c598b7-f86vh\" (UID: \"22545d04-38b5-4d64-89a0-9e533875220e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c92aec-5bd6c598b7-f86vh" Oct 30 22:26:03.558984 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:03.558952 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c92aec-5bd6c598b7-f86vh" Oct 30 22:26:03.695168 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:03.694566 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:03.849758 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:03.849722 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c92aec-5bd6c598b7-f86vh"] Oct 30 22:26:03.851667 ip-10-0-137-181 kubenswrapper[6720]: W1030 22:26:03.851639 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22545d04_38b5_4d64_89a0_9e533875220e.slice/crio-9c072aba05841381dee38d644409505ce32dbf4e3ae16d0ac82f13d46ca9ae30 WatchSource:0}: Error finding container 9c072aba05841381dee38d644409505ce32dbf4e3ae16d0ac82f13d46ca9ae30: Status 404 returned error can't find the container with id 9c072aba05841381dee38d644409505ce32dbf4e3ae16d0ac82f13d46ca9ae30 Oct 30 22:26:04.180852 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:04.180805 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c92aec-5bd6c598b7-f86vh" event={"ID":"22545d04-38b5-4d64-89a0-9e533875220e","Type":"ContainerStarted","Data":"9c072aba05841381dee38d644409505ce32dbf4e3ae16d0ac82f13d46ca9ae30"} Oct 30 22:26:04.378191 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:04.378156 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:05.062677 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:05.062640 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:05.188481 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:05.188442 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c92aec-5bd6c598b7-f86vh" event={"ID":"22545d04-38b5-4d64-89a0-9e533875220e","Type":"ContainerStarted","Data":"4aa532de7b40bd6d0e1e14da80f5f23efed42aa70402a94491e4bc801be879b0"} Oct 30 22:26:05.188906 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:05.188621 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c92aec-5bd6c598b7-f86vh" Oct 30 22:26:05.212845 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:05.212786 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c92aec-5bd6c598b7-f86vh" podStartSLOduration=2.769759032 podStartE2EDuration="3.212767896s" totalImagesPullingTime="443.008864ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:05.209023914 +0000 UTC m=+2731.379989097" watchObservedRunningTime="2026-10-30 22:26:05.212767896 +0000 UTC m=+2731.383733082" Oct 30 22:26:05.719068 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:05.719028 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:06.633558 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:06.633524 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:07.354016 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:07.353974 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:08.031839 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:08.031800 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:08.180563 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:08.180534 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b2bb0-ffd47c566-qwcvz" Oct 30 22:26:08.815962 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:08.815076 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:09.445724 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:09.445668 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:10.128866 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:10.128825 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:10.403748 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:26:10.403663 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 22:26:10.693942 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:10.693862 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-d8c835-848b64f74c-kn8h6"] Oct 30 22:26:10.694662 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:10.694633 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d8c835-848b64f74c-kn8h6" podUID="ca63c13b-3454-470e-80a0-d59435e6533f" containerName="maas-api" containerID="cri-o://e93c564228de69eb6f83bd95c8127739133cf8f0e059e8e73f4e796ba48ad415" gracePeriod=30 Oct 30 22:26:10.992078 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:10.991998 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:11.203852 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:11.203824 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c92aec-5bd6c598b7-f86vh" Oct 30 22:26:11.242829 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:11.242798 6720 generic.go:356] "Generic (PLEG): container finished" podID="ca63c13b-3454-470e-80a0-d59435e6533f" containerID="e93c564228de69eb6f83bd95c8127739133cf8f0e059e8e73f4e796ba48ad415" exitCode=0 Oct 30 22:26:11.243054 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:11.242884 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d8c835-848b64f74c-kn8h6" event={"ID":"ca63c13b-3454-470e-80a0-d59435e6533f","Type":"ContainerDied","Data":"e93c564228de69eb6f83bd95c8127739133cf8f0e059e8e73f4e796ba48ad415"} Oct 30 22:26:11.243054 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:11.242935 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d8c835-848b64f74c-kn8h6" event={"ID":"ca63c13b-3454-470e-80a0-d59435e6533f","Type":"ContainerDied","Data":"cfccf890722f9261ebc9a141eec6fbc3190dc7c696c3db4e1712ebc6af5c24ef"} Oct 30 22:26:11.243054 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:11.242954 6720 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cfccf890722f9261ebc9a141eec6fbc3190dc7c696c3db4e1712ebc6af5c24ef" Oct 30 22:26:11.243185 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:11.243154 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d8c835-848b64f74c-kn8h6" Oct 30 22:26:11.280062 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:11.280031 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:11.295807 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:11.295768 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca63c13b-3454-470e-80a0-d59435e6533f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca63c13b-3454-470e-80a0-d59435e6533f-maas-api-tls\") pod \"ca63c13b-3454-470e-80a0-d59435e6533f\" (UID: \"ca63c13b-3454-470e-80a0-d59435e6533f\") " Oct 30 22:26:11.296010 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:11.295834 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca63c13b-3454-470e-80a0-d59435e6533f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca63c13b-3454-470e-80a0-d59435e6533f-openshift-service-ca\") pod \"ca63c13b-3454-470e-80a0-d59435e6533f\" (UID: \"ca63c13b-3454-470e-80a0-d59435e6533f\") " Oct 30 22:26:11.296010 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:11.295939 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca63c13b-3454-470e-80a0-d59435e6533f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca63c13b-3454-470e-80a0-d59435e6533f-metrics-certs\") pod \"ca63c13b-3454-470e-80a0-d59435e6533f\" (UID: \"ca63c13b-3454-470e-80a0-d59435e6533f\") " Oct 30 22:26:11.296323 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:11.296291 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca63c13b-3454-470e-80a0-d59435e6533f-openshift-service-ca" pod "ca63c13b-3454-470e-80a0-d59435e6533f" (UID: "ca63c13b-3454-470e-80a0-d59435e6533f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:26:11.296533 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:11.296509 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca63c13b-3454-470e-80a0-d59435e6533f-kube-api-access-w24xg\" (UniqueName: \"kubernetes.io/projected/ca63c13b-3454-470e-80a0-d59435e6533f-kube-api-access-w24xg\") pod \"ca63c13b-3454-470e-80a0-d59435e6533f\" (UID: \"ca63c13b-3454-470e-80a0-d59435e6533f\") " Oct 30 22:26:11.297595 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:11.297571 6720 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca63c13b-3454-470e-80a0-d59435e6533f-openshift-service-ca\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:11.298996 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:11.298961 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca63c13b-3454-470e-80a0-d59435e6533f-kube-api-access-w24xg" pod "ca63c13b-3454-470e-80a0-d59435e6533f" (UID: "ca63c13b-3454-470e-80a0-d59435e6533f"). InnerVolumeSpecName "kube-api-access-w24xg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:26:11.309542 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:11.309519 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca63c13b-3454-470e-80a0-d59435e6533f-maas-api-tls" pod "ca63c13b-3454-470e-80a0-d59435e6533f" (UID: "ca63c13b-3454-470e-80a0-d59435e6533f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:26:11.309774 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:11.309749 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca63c13b-3454-470e-80a0-d59435e6533f-metrics-certs" pod "ca63c13b-3454-470e-80a0-d59435e6533f" (UID: "ca63c13b-3454-470e-80a0-d59435e6533f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:26:11.405283 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:11.405245 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w24xg\" (UniqueName: \"kubernetes.io/projected/ca63c13b-3454-470e-80a0-d59435e6533f-kube-api-access-w24xg\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:11.405283 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:11.405284 6720 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ca63c13b-3454-470e-80a0-d59435e6533f-maas-api-tls\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:11.405616 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:11.405300 6720 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ca63c13b-3454-470e-80a0-d59435e6533f-metrics-certs\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:11.433892 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:11.433855 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:11.827440 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:11.827407 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:12.249198 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:12.249167 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-d8c835-848b64f74c-kn8h6" Oct 30 22:26:12.270121 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:12.270088 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-d8c835-848b64f74c-kn8h6"] Oct 30 22:26:12.276052 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:12.276026 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-d8c835-848b64f74c-kn8h6"] Oct 30 22:26:12.328280 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:12.315953 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:12.828856 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:12.828821 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-x64z9"] Oct 30 22:26:12.829836 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:12.829807 6720 cpu_manager.go:414] "containerMap: removing container" podUID="ca63c13b-3454-470e-80a0-d59435e6533f" containerName="maas-api" Oct 30 22:26:12.829836 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:12.829833 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca63c13b-3454-470e-80a0-d59435e6533f" containerName="maas-api" Oct 30 22:26:12.830072 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:12.830058 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca63c13b-3454-470e-80a0-d59435e6533f" containerName="maas-api" Oct 30 22:26:12.834313 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:12.834290 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-x64z9" Oct 30 22:26:12.852442 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:12.852412 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-x64z9"] Oct 30 22:26:12.919575 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:12.919542 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-120cb1-openshift-default-dockercfg-9p94d\"" Oct 30 22:26:12.928553 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:12.928523 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kj84j\" (UniqueName: \"kubernetes.io/projected/ea5efed8-2dea-46bf-9162-55fe6a1459d5-kube-api-access-kj84j\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-x64z9\" (UID: \"ea5efed8-2dea-46bf-9162-55fe6a1459d5\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-x64z9" Oct 30 22:26:12.928707 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:12.928581 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea5efed8-2dea-46bf-9162-55fe6a1459d5-workload-socket\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-x64z9\" (UID: \"ea5efed8-2dea-46bf-9162-55fe6a1459d5\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-x64z9" Oct 30 22:26:12.928760 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:12.928712 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea5efed8-2dea-46bf-9162-55fe6a1459d5-istio-data\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-x64z9\" (UID: \"ea5efed8-2dea-46bf-9162-55fe6a1459d5\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-x64z9" Oct 30 22:26:12.928799 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:12.928758 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea5efed8-2dea-46bf-9162-55fe6a1459d5-workload-certs\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-x64z9\" (UID: \"ea5efed8-2dea-46bf-9162-55fe6a1459d5\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-x64z9" Oct 30 22:26:12.928840 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:12.928823 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea5efed8-2dea-46bf-9162-55fe6a1459d5-istio-podinfo\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-x64z9\" (UID: \"ea5efed8-2dea-46bf-9162-55fe6a1459d5\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-x64z9" Oct 30 22:26:12.928952 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:12.928939 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea5efed8-2dea-46bf-9162-55fe6a1459d5-istio-token\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-x64z9\" (UID: \"ea5efed8-2dea-46bf-9162-55fe6a1459d5\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-x64z9" Oct 30 22:26:12.928992 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:12.928969 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea5efed8-2dea-46bf-9162-55fe6a1459d5-credential-socket\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-x64z9\" (UID: \"ea5efed8-2dea-46bf-9162-55fe6a1459d5\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-x64z9" Oct 30 22:26:12.929024 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:12.928989 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea5efed8-2dea-46bf-9162-55fe6a1459d5-istio-envoy\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-x64z9\" (UID: \"ea5efed8-2dea-46bf-9162-55fe6a1459d5\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-x64z9" Oct 30 22:26:12.929060 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:12.929051 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea5efed8-2dea-46bf-9162-55fe6a1459d5-istiod-ca-cert\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-x64z9\" (UID: \"ea5efed8-2dea-46bf-9162-55fe6a1459d5\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-x64z9" Oct 30 22:26:13.034485 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:13.032590 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea5efed8-2dea-46bf-9162-55fe6a1459d5-workload-socket\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-x64z9\" (UID: \"ea5efed8-2dea-46bf-9162-55fe6a1459d5\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-x64z9" Oct 30 22:26:13.034485 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:13.032713 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea5efed8-2dea-46bf-9162-55fe6a1459d5-istio-data\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-x64z9\" (UID: \"ea5efed8-2dea-46bf-9162-55fe6a1459d5\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-x64z9" Oct 30 22:26:13.034485 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:13.032772 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea5efed8-2dea-46bf-9162-55fe6a1459d5-workload-certs\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-x64z9\" (UID: \"ea5efed8-2dea-46bf-9162-55fe6a1459d5\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-x64z9" Oct 30 22:26:13.034485 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:13.032804 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea5efed8-2dea-46bf-9162-55fe6a1459d5-istio-podinfo\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-x64z9\" (UID: \"ea5efed8-2dea-46bf-9162-55fe6a1459d5\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-x64z9" Oct 30 22:26:13.034485 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:13.032877 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea5efed8-2dea-46bf-9162-55fe6a1459d5-istio-token\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-x64z9\" (UID: \"ea5efed8-2dea-46bf-9162-55fe6a1459d5\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-x64z9" Oct 30 22:26:13.034485 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:13.032916 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea5efed8-2dea-46bf-9162-55fe6a1459d5-credential-socket\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-x64z9\" (UID: \"ea5efed8-2dea-46bf-9162-55fe6a1459d5\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-x64z9" Oct 30 22:26:13.034485 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:13.032943 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea5efed8-2dea-46bf-9162-55fe6a1459d5-istio-envoy\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-x64z9\" (UID: \"ea5efed8-2dea-46bf-9162-55fe6a1459d5\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-x64z9" Oct 30 22:26:13.034485 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:13.032988 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea5efed8-2dea-46bf-9162-55fe6a1459d5-istiod-ca-cert\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-x64z9\" (UID: \"ea5efed8-2dea-46bf-9162-55fe6a1459d5\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-x64z9" Oct 30 22:26:13.034485 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:13.033111 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj84j\" (UniqueName: \"kubernetes.io/projected/ea5efed8-2dea-46bf-9162-55fe6a1459d5-kube-api-access-kj84j\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-x64z9\" (UID: \"ea5efed8-2dea-46bf-9162-55fe6a1459d5\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-x64z9" Oct 30 22:26:13.034485 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:13.033885 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea5efed8-2dea-46bf-9162-55fe6a1459d5-workload-socket\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-x64z9\" (UID: \"ea5efed8-2dea-46bf-9162-55fe6a1459d5\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-x64z9" Oct 30 22:26:13.034485 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:13.034126 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea5efed8-2dea-46bf-9162-55fe6a1459d5-credential-socket\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-x64z9\" (UID: \"ea5efed8-2dea-46bf-9162-55fe6a1459d5\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-x64z9" Oct 30 22:26:13.034485 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:13.034432 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea5efed8-2dea-46bf-9162-55fe6a1459d5-istio-data\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-x64z9\" (UID: \"ea5efed8-2dea-46bf-9162-55fe6a1459d5\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-x64z9" Oct 30 22:26:13.035298 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:13.034664 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea5efed8-2dea-46bf-9162-55fe6a1459d5-workload-certs\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-x64z9\" (UID: \"ea5efed8-2dea-46bf-9162-55fe6a1459d5\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-x64z9" Oct 30 22:26:13.035298 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:13.035058 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea5efed8-2dea-46bf-9162-55fe6a1459d5-istiod-ca-cert\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-x64z9\" (UID: \"ea5efed8-2dea-46bf-9162-55fe6a1459d5\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-x64z9" Oct 30 22:26:13.048620 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:13.040801 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea5efed8-2dea-46bf-9162-55fe6a1459d5-istio-envoy\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-x64z9\" (UID: \"ea5efed8-2dea-46bf-9162-55fe6a1459d5\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-x64z9" Oct 30 22:26:13.048620 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:13.040981 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea5efed8-2dea-46bf-9162-55fe6a1459d5-istio-podinfo\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-x64z9\" (UID: \"ea5efed8-2dea-46bf-9162-55fe6a1459d5\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-x64z9" Oct 30 22:26:13.059232 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:13.059183 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kj84j\" (UniqueName: \"kubernetes.io/projected/ea5efed8-2dea-46bf-9162-55fe6a1459d5-kube-api-access-kj84j\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-x64z9\" (UID: \"ea5efed8-2dea-46bf-9162-55fe6a1459d5\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-x64z9" Oct 30 22:26:13.060661 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:13.060632 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea5efed8-2dea-46bf-9162-55fe6a1459d5-istio-token\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-x64z9\" (UID: \"ea5efed8-2dea-46bf-9162-55fe6a1459d5\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-x64z9" Oct 30 22:26:13.078512 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:13.078485 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:13.155839 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:13.155809 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-x64z9" Oct 30 22:26:13.447794 ip-10-0-137-181 kubenswrapper[6720]: W1030 22:26:13.447763 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea5efed8_2dea_46bf_9162_55fe6a1459d5.slice/crio-6c3e80a149f0c385a74fc7a52127be22089f365b8b331d5a3c233c54703f6d8d WatchSource:0}: Error finding container 6c3e80a149f0c385a74fc7a52127be22089f365b8b331d5a3c233c54703f6d8d: Status 404 returned error can't find the container with id 6c3e80a149f0c385a74fc7a52127be22089f365b8b331d5a3c233c54703f6d8d Oct 30 22:26:13.447904 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:13.447875 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-x64z9"] Oct 30 22:26:13.450733 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:13.450689 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:26:13.450836 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:13.450768 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:26:13.450836 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:13.450797 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:26:13.927412 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:13.922736 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-xbbpj"] Oct 30 22:26:13.957620 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:13.957576 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-xbbpj" Oct 30 22:26:13.965374 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:13.964736 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:14.002747 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:14.002709 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-df200a-openshift-default-dockercfg-rrfqx\"" Oct 30 22:26:14.011307 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:14.011275 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca63c13b-3454-470e-80a0-d59435e6533f" path="/var/lib/kubelet/pods/ca63c13b-3454-470e-80a0-d59435e6533f/volumes" Oct 30 22:26:14.011629 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:14.011614 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-xbbpj"] Oct 30 22:26:14.073467 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:14.069327 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47d2f169-c1d8-4533-be16-9d195a7f73c9-istio-token\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-xbbpj\" (UID: \"47d2f169-c1d8-4533-be16-9d195a7f73c9\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-xbbpj" Oct 30 22:26:14.073467 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:14.069403 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-626xp\" (UniqueName: \"kubernetes.io/projected/47d2f169-c1d8-4533-be16-9d195a7f73c9-kube-api-access-626xp\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-xbbpj\" (UID: \"47d2f169-c1d8-4533-be16-9d195a7f73c9\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-xbbpj" Oct 30 22:26:14.073467 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:14.069442 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47d2f169-c1d8-4533-be16-9d195a7f73c9-istiod-ca-cert\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-xbbpj\" (UID: \"47d2f169-c1d8-4533-be16-9d195a7f73c9\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-xbbpj" Oct 30 22:26:14.073467 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:14.069502 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d2f169-c1d8-4533-be16-9d195a7f73c9-workload-socket\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-xbbpj\" (UID: \"47d2f169-c1d8-4533-be16-9d195a7f73c9\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-xbbpj" Oct 30 22:26:14.073467 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:14.069624 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47d2f169-c1d8-4533-be16-9d195a7f73c9-istio-podinfo\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-xbbpj\" (UID: \"47d2f169-c1d8-4533-be16-9d195a7f73c9\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-xbbpj" Oct 30 22:26:14.073467 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:14.069752 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47d2f169-c1d8-4533-be16-9d195a7f73c9-istio-data\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-xbbpj\" (UID: \"47d2f169-c1d8-4533-be16-9d195a7f73c9\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-xbbpj" Oct 30 22:26:14.073467 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:14.069807 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d2f169-c1d8-4533-be16-9d195a7f73c9-credential-socket\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-xbbpj\" (UID: \"47d2f169-c1d8-4533-be16-9d195a7f73c9\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-xbbpj" Oct 30 22:26:14.073467 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:14.069844 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47d2f169-c1d8-4533-be16-9d195a7f73c9-workload-certs\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-xbbpj\" (UID: \"47d2f169-c1d8-4533-be16-9d195a7f73c9\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-xbbpj" Oct 30 22:26:14.073467 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:14.070021 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47d2f169-c1d8-4533-be16-9d195a7f73c9-istio-envoy\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-xbbpj\" (UID: \"47d2f169-c1d8-4533-be16-9d195a7f73c9\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-xbbpj" Oct 30 22:26:14.171021 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:14.170991 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47d2f169-c1d8-4533-be16-9d195a7f73c9-istio-data\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-xbbpj\" (UID: \"47d2f169-c1d8-4533-be16-9d195a7f73c9\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-xbbpj" Oct 30 22:26:14.171225 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:14.171039 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d2f169-c1d8-4533-be16-9d195a7f73c9-credential-socket\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-xbbpj\" (UID: \"47d2f169-c1d8-4533-be16-9d195a7f73c9\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-xbbpj" Oct 30 22:26:14.171225 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:14.171176 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47d2f169-c1d8-4533-be16-9d195a7f73c9-workload-certs\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-xbbpj\" (UID: \"47d2f169-c1d8-4533-be16-9d195a7f73c9\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-xbbpj" Oct 30 22:26:14.171348 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:14.171306 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47d2f169-c1d8-4533-be16-9d195a7f73c9-istio-envoy\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-xbbpj\" (UID: \"47d2f169-c1d8-4533-be16-9d195a7f73c9\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-xbbpj" Oct 30 22:26:14.171509 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:14.171486 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47d2f169-c1d8-4533-be16-9d195a7f73c9-istio-data\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-xbbpj\" (UID: \"47d2f169-c1d8-4533-be16-9d195a7f73c9\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-xbbpj" Oct 30 22:26:14.171509 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:14.171500 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d2f169-c1d8-4533-be16-9d195a7f73c9-credential-socket\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-xbbpj\" (UID: \"47d2f169-c1d8-4533-be16-9d195a7f73c9\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-xbbpj" Oct 30 22:26:14.171647 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:14.171598 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47d2f169-c1d8-4533-be16-9d195a7f73c9-istio-token\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-xbbpj\" (UID: \"47d2f169-c1d8-4533-be16-9d195a7f73c9\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-xbbpj" Oct 30 22:26:14.171647 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:14.171629 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-626xp\" (UniqueName: \"kubernetes.io/projected/47d2f169-c1d8-4533-be16-9d195a7f73c9-kube-api-access-626xp\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-xbbpj\" (UID: \"47d2f169-c1d8-4533-be16-9d195a7f73c9\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-xbbpj" Oct 30 22:26:14.171763 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:14.171662 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47d2f169-c1d8-4533-be16-9d195a7f73c9-istiod-ca-cert\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-xbbpj\" (UID: \"47d2f169-c1d8-4533-be16-9d195a7f73c9\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-xbbpj" Oct 30 22:26:14.171763 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:14.171725 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d2f169-c1d8-4533-be16-9d195a7f73c9-workload-socket\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-xbbpj\" (UID: \"47d2f169-c1d8-4533-be16-9d195a7f73c9\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-xbbpj" Oct 30 22:26:14.171865 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:14.171756 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47d2f169-c1d8-4533-be16-9d195a7f73c9-workload-certs\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-xbbpj\" (UID: \"47d2f169-c1d8-4533-be16-9d195a7f73c9\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-xbbpj" Oct 30 22:26:14.172073 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:14.172049 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d2f169-c1d8-4533-be16-9d195a7f73c9-workload-socket\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-xbbpj\" (UID: \"47d2f169-c1d8-4533-be16-9d195a7f73c9\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-xbbpj" Oct 30 22:26:14.172164 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:14.171845 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47d2f169-c1d8-4533-be16-9d195a7f73c9-istio-podinfo\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-xbbpj\" (UID: \"47d2f169-c1d8-4533-be16-9d195a7f73c9\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-xbbpj" Oct 30 22:26:14.172708 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:14.172684 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47d2f169-c1d8-4533-be16-9d195a7f73c9-istiod-ca-cert\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-xbbpj\" (UID: \"47d2f169-c1d8-4533-be16-9d195a7f73c9\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-xbbpj" Oct 30 22:26:14.174204 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:14.174182 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47d2f169-c1d8-4533-be16-9d195a7f73c9-istio-envoy\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-xbbpj\" (UID: \"47d2f169-c1d8-4533-be16-9d195a7f73c9\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-xbbpj" Oct 30 22:26:14.174975 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:14.174936 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47d2f169-c1d8-4533-be16-9d195a7f73c9-istio-podinfo\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-xbbpj\" (UID: \"47d2f169-c1d8-4533-be16-9d195a7f73c9\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-xbbpj" Oct 30 22:26:14.185251 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:14.185163 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47d2f169-c1d8-4533-be16-9d195a7f73c9-istio-token\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-xbbpj\" (UID: \"47d2f169-c1d8-4533-be16-9d195a7f73c9\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-xbbpj" Oct 30 22:26:14.185986 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:14.185674 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-626xp\" (UniqueName: \"kubernetes.io/projected/47d2f169-c1d8-4533-be16-9d195a7f73c9-kube-api-access-626xp\") pod \"e2e-inf-a-df200a-openshift-default-86d8f88f8f-xbbpj\" (UID: \"47d2f169-c1d8-4533-be16-9d195a7f73c9\") " pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-xbbpj" Oct 30 22:26:14.317467 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:14.308844 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-xbbpj" Oct 30 22:26:14.331972 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:14.331924 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-x64z9" event={"ID":"ea5efed8-2dea-46bf-9162-55fe6a1459d5","Type":"ContainerStarted","Data":"186fdb74a0368dc7020f7e17c2afefa3e66b060f9f47ec8464d20ee09504852c"} Oct 30 22:26:14.332124 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:14.332015 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-x64z9" event={"ID":"ea5efed8-2dea-46bf-9162-55fe6a1459d5","Type":"ContainerStarted","Data":"6c3e80a149f0c385a74fc7a52127be22089f365b8b331d5a3c233c54703f6d8d"} Oct 30 22:26:14.418414 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:14.416782 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-x64z9" podStartSLOduration=2.416761841 podStartE2EDuration="2.416761841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:14.414435989 +0000 UTC m=+2740.585401167" watchObservedRunningTime="2026-10-30 22:26:14.416761841 +0000 UTC m=+2740.587727025" Oct 30 22:26:14.691920 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:14.691828 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-xbbpj"] Oct 30 22:26:14.703608 ip-10-0-137-181 kubenswrapper[6720]: W1030 22:26:14.703517 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47d2f169_c1d8_4533_be16_9d195a7f73c9.slice/crio-f5e1b800305456071e237f3b3bcf27ad80aa68e9ca90ca6860c32932be4deef5 WatchSource:0}: Error finding container f5e1b800305456071e237f3b3bcf27ad80aa68e9ca90ca6860c32932be4deef5: Status 404 returned error can't find the container with id f5e1b800305456071e237f3b3bcf27ad80aa68e9ca90ca6860c32932be4deef5 Oct 30 22:26:14.713657 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:14.713613 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:26:14.713752 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:14.713711 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:26:14.713791 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:14.713751 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:26:15.075553 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:15.075436 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:15.156663 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:15.156578 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-x64z9" Oct 30 22:26:15.158658 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:15.158626 6720 patch_prober.go:29] interesting pod/e2e-inf-a-120cb1-openshift-default-d97c644bf-x64z9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 30 22:26:15.158932 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:15.158904 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-x64z9" podUID="ea5efed8-2dea-46bf-9162-55fe6a1459d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 30 22:26:15.340834 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:15.340752 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-xbbpj" event={"ID":"47d2f169-c1d8-4533-be16-9d195a7f73c9","Type":"ContainerStarted","Data":"713c79240f39a70ec0bbe2856a82666fd7cab78db9cfd6cbc110498580f9a2c8"} Oct 30 22:26:15.340834 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:15.340810 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-xbbpj" event={"ID":"47d2f169-c1d8-4533-be16-9d195a7f73c9","Type":"ContainerStarted","Data":"f5e1b800305456071e237f3b3bcf27ad80aa68e9ca90ca6860c32932be4deef5"} Oct 30 22:26:15.374202 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:15.374140 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-xbbpj" podStartSLOduration=2.374120187 podStartE2EDuration="2.374120187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:15.372515634 +0000 UTC m=+2741.543480836" watchObservedRunningTime="2026-10-30 22:26:15.374120187 +0000 UTC m=+2741.545085370" Oct 30 22:26:16.094458 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:16.092916 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:16.165412 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:16.164835 6720 patch_prober.go:29] interesting pod/e2e-inf-a-120cb1-openshift-default-d97c644bf-x64z9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 30 22:26:16.165412 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:16.164907 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-x64z9" podUID="ea5efed8-2dea-46bf-9162-55fe6a1459d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 30 22:26:16.309954 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:16.309878 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-xbbpj" Oct 30 22:26:16.310935 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:16.310914 6720 patch_prober.go:29] interesting pod/e2e-inf-a-df200a-openshift-default-86d8f88f8f-xbbpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Oct 30 22:26:16.311005 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:16.310960 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-xbbpj" podUID="47d2f169-c1d8-4533-be16-9d195a7f73c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Oct 30 22:26:16.911305 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:16.906474 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:17.164471 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:17.163820 6720 patch_prober.go:29] interesting pod/e2e-inf-a-120cb1-openshift-default-d97c644bf-x64z9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 30 22:26:17.164471 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:17.163900 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-x64z9" podUID="ea5efed8-2dea-46bf-9162-55fe6a1459d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 30 22:26:17.309969 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:17.309934 6720 patch_prober.go:29] interesting pod/e2e-inf-a-df200a-openshift-default-86d8f88f8f-xbbpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Oct 30 22:26:17.310153 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:17.310006 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-xbbpj" podUID="47d2f169-c1d8-4533-be16-9d195a7f73c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Oct 30 22:26:18.177411 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:18.165956 6720 patch_prober.go:29] interesting pod/e2e-inf-a-120cb1-openshift-default-d97c644bf-x64z9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 30 22:26:18.177411 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:18.166016 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-x64z9" podUID="ea5efed8-2dea-46bf-9162-55fe6a1459d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 30 22:26:18.177411 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:18.170909 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:18.309760 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:18.309728 6720 patch_prober.go:29] interesting pod/e2e-inf-a-df200a-openshift-default-86d8f88f8f-xbbpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Oct 30 22:26:18.309932 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:18.309806 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-xbbpj" podUID="47d2f169-c1d8-4533-be16-9d195a7f73c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Oct 30 22:26:19.185412 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:19.179566 6720 patch_prober.go:29] interesting pod/e2e-inf-a-120cb1-openshift-default-d97c644bf-x64z9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 30 22:26:19.185412 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:19.179630 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-x64z9" podUID="ea5efed8-2dea-46bf-9162-55fe6a1459d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 30 22:26:19.255499 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:19.252808 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:19.323418 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:19.314514 6720 patch_prober.go:29] interesting pod/e2e-inf-a-df200a-openshift-default-86d8f88f8f-xbbpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Oct 30 22:26:19.323418 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:19.314573 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-xbbpj" podUID="47d2f169-c1d8-4533-be16-9d195a7f73c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Oct 30 22:26:19.431057 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:19.429706 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:19.622667 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:19.622625 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:20.202759 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:20.202735 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-x64z9" Oct 30 22:26:20.203999 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:20.203835 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-x64z9" Oct 30 22:26:20.204608 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:20.204590 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-x64z9" Oct 30 22:26:20.319753 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:20.319526 6720 patch_prober.go:29] interesting pod/e2e-inf-a-df200a-openshift-default-86d8f88f8f-xbbpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Oct 30 22:26:20.319753 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:20.319598 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-xbbpj" podUID="47d2f169-c1d8-4533-be16-9d195a7f73c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Oct 30 22:26:20.613754 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:20.608428 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:21.331407 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:21.330465 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-xbbpj" Oct 30 22:26:21.331898 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:21.331884 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-xbbpj" Oct 30 22:26:21.333169 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:21.333142 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-xbbpj" Oct 30 22:26:21.612138 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:21.612060 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:22.238666 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:22.238595 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:22.832483 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:22.832446 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-58d5fa-58b5b7b7df-nzxwm"] Oct 30 22:26:22.837079 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:22.837046 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58d5fa-58b5b7b7df-nzxwm" Oct 30 22:26:22.848446 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:22.848366 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-58d5fa-serving-cert\"" Oct 30 22:26:22.857098 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:22.857065 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-58d5fa-58b5b7b7df-nzxwm"] Oct 30 22:26:22.870324 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:22.870284 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6c2aa19-cad5-435a-a573-56146651be06-openshift-service-ca\") pod \"maas-api-e2e-inf-b-58d5fa-58b5b7b7df-nzxwm\" (UID: \"b6c2aa19-cad5-435a-a573-56146651be06\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58d5fa-58b5b7b7df-nzxwm" Oct 30 22:26:22.870525 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:22.870371 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n475c\" (UniqueName: \"kubernetes.io/projected/b6c2aa19-cad5-435a-a573-56146651be06-kube-api-access-n475c\") pod \"maas-api-e2e-inf-b-58d5fa-58b5b7b7df-nzxwm\" (UID: \"b6c2aa19-cad5-435a-a573-56146651be06\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58d5fa-58b5b7b7df-nzxwm" Oct 30 22:26:22.870597 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:22.870560 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6c2aa19-cad5-435a-a573-56146651be06-maas-api-tls\") pod \"maas-api-e2e-inf-b-58d5fa-58b5b7b7df-nzxwm\" (UID: \"b6c2aa19-cad5-435a-a573-56146651be06\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58d5fa-58b5b7b7df-nzxwm" Oct 30 22:26:22.870793 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:22.870770 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6c2aa19-cad5-435a-a573-56146651be06-metrics-certs\") pod \"maas-api-e2e-inf-b-58d5fa-58b5b7b7df-nzxwm\" (UID: \"b6c2aa19-cad5-435a-a573-56146651be06\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58d5fa-58b5b7b7df-nzxwm" Oct 30 22:26:22.937637 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:22.937603 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:22.978998 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:22.978947 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6c2aa19-cad5-435a-a573-56146651be06-metrics-certs\") pod \"maas-api-e2e-inf-b-58d5fa-58b5b7b7df-nzxwm\" (UID: \"b6c2aa19-cad5-435a-a573-56146651be06\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58d5fa-58b5b7b7df-nzxwm" Oct 30 22:26:22.979983 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:22.979953 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6c2aa19-cad5-435a-a573-56146651be06-openshift-service-ca\") pod \"maas-api-e2e-inf-b-58d5fa-58b5b7b7df-nzxwm\" (UID: \"b6c2aa19-cad5-435a-a573-56146651be06\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58d5fa-58b5b7b7df-nzxwm" Oct 30 22:26:22.981220 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:22.981198 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n475c\" (UniqueName: \"kubernetes.io/projected/b6c2aa19-cad5-435a-a573-56146651be06-kube-api-access-n475c\") pod \"maas-api-e2e-inf-b-58d5fa-58b5b7b7df-nzxwm\" (UID: \"b6c2aa19-cad5-435a-a573-56146651be06\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58d5fa-58b5b7b7df-nzxwm" Oct 30 22:26:22.985944 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:22.985915 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6c2aa19-cad5-435a-a573-56146651be06-maas-api-tls\") pod \"maas-api-e2e-inf-b-58d5fa-58b5b7b7df-nzxwm\" (UID: \"b6c2aa19-cad5-435a-a573-56146651be06\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58d5fa-58b5b7b7df-nzxwm" Oct 30 22:26:22.986159 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:22.981043 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6c2aa19-cad5-435a-a573-56146651be06-openshift-service-ca\") pod \"maas-api-e2e-inf-b-58d5fa-58b5b7b7df-nzxwm\" (UID: \"b6c2aa19-cad5-435a-a573-56146651be06\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58d5fa-58b5b7b7df-nzxwm" Oct 30 22:26:22.986911 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:26:22.986880 6720 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-58d5fa-serving-cert: secret "maas-api-e2e-inf-b-58d5fa-serving-cert" not found Oct 30 22:26:22.987142 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:26:22.987127 6720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6c2aa19-cad5-435a-a573-56146651be06-maas-api-tls podName:b6c2aa19-cad5-435a-a573-56146651be06 nodeName:}" failed. No retries permitted until 2026-10-30 22:26:23.4871002 +0000 UTC m=+2749.658065378 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b6c2aa19-cad5-435a-a573-56146651be06-maas-api-tls") pod "maas-api-e2e-inf-b-58d5fa-58b5b7b7df-nzxwm" (UID: "b6c2aa19-cad5-435a-a573-56146651be06") : secret "maas-api-e2e-inf-b-58d5fa-serving-cert" not found Oct 30 22:26:22.993621 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:22.991175 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6c2aa19-cad5-435a-a573-56146651be06-metrics-certs\") pod \"maas-api-e2e-inf-b-58d5fa-58b5b7b7df-nzxwm\" (UID: \"b6c2aa19-cad5-435a-a573-56146651be06\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58d5fa-58b5b7b7df-nzxwm" Oct 30 22:26:23.003957 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:23.003541 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n475c\" (UniqueName: \"kubernetes.io/projected/b6c2aa19-cad5-435a-a573-56146651be06-kube-api-access-n475c\") pod \"maas-api-e2e-inf-b-58d5fa-58b5b7b7df-nzxwm\" (UID: \"b6c2aa19-cad5-435a-a573-56146651be06\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58d5fa-58b5b7b7df-nzxwm" Oct 30 22:26:23.086870 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:23.086785 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:23.501410 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:23.498642 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6c2aa19-cad5-435a-a573-56146651be06-maas-api-tls\") pod \"maas-api-e2e-inf-b-58d5fa-58b5b7b7df-nzxwm\" (UID: \"b6c2aa19-cad5-435a-a573-56146651be06\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58d5fa-58b5b7b7df-nzxwm" Oct 30 22:26:23.501410 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:26:23.498898 6720 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-58d5fa-serving-cert: secret "maas-api-e2e-inf-b-58d5fa-serving-cert" not found Oct 30 22:26:23.501410 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:26:23.498969 6720 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6c2aa19-cad5-435a-a573-56146651be06-maas-api-tls podName:b6c2aa19-cad5-435a-a573-56146651be06 nodeName:}" failed. No retries permitted until 2026-10-30 22:26:24.498949777 +0000 UTC m=+2750.669914959 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b6c2aa19-cad5-435a-a573-56146651be06-maas-api-tls") pod "maas-api-e2e-inf-b-58d5fa-58b5b7b7df-nzxwm" (UID: "b6c2aa19-cad5-435a-a573-56146651be06") : secret "maas-api-e2e-inf-b-58d5fa-serving-cert" not found Oct 30 22:26:23.936580 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:23.936531 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:24.515892 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:24.515849 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6c2aa19-cad5-435a-a573-56146651be06-maas-api-tls\") pod \"maas-api-e2e-inf-b-58d5fa-58b5b7b7df-nzxwm\" (UID: \"b6c2aa19-cad5-435a-a573-56146651be06\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58d5fa-58b5b7b7df-nzxwm" Oct 30 22:26:24.518316 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:24.518290 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:24.518757 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:24.518738 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6c2aa19-cad5-435a-a573-56146651be06-maas-api-tls\") pod \"maas-api-e2e-inf-b-58d5fa-58b5b7b7df-nzxwm\" (UID: \"b6c2aa19-cad5-435a-a573-56146651be06\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58d5fa-58b5b7b7df-nzxwm" Oct 30 22:26:24.656870 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:24.656831 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58d5fa-58b5b7b7df-nzxwm" Oct 30 22:26:24.776456 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:24.776153 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:24.949024 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:24.948995 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-58d5fa-58b5b7b7df-nzxwm"] Oct 30 22:26:24.951867 ip-10-0-137-181 kubenswrapper[6720]: W1030 22:26:24.951836 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6c2aa19_cad5_435a_a573_56146651be06.slice/crio-13ab8eca3f040397cf61ddf50fa6759540691faadd4fdddf78f37afac134649a WatchSource:0}: Error finding container 13ab8eca3f040397cf61ddf50fa6759540691faadd4fdddf78f37afac134649a: Status 404 returned error can't find the container with id 13ab8eca3f040397cf61ddf50fa6759540691faadd4fdddf78f37afac134649a Oct 30 22:26:25.518751 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:25.518710 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58d5fa-58b5b7b7df-nzxwm" event={"ID":"b6c2aa19-cad5-435a-a573-56146651be06","Type":"ContainerStarted","Data":"13ab8eca3f040397cf61ddf50fa6759540691faadd4fdddf78f37afac134649a"} Oct 30 22:26:25.603419 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:25.602123 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:26.521802 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:26.521079 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:26.549335 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:26.549283 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58d5fa-58b5b7b7df-nzxwm" event={"ID":"b6c2aa19-cad5-435a-a573-56146651be06","Type":"ContainerStarted","Data":"e974e2b2a84cb8b12c42b612230e51f70f231ae888cb9efad5226d1bcffe7769"} Oct 30 22:26:26.549601 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:26.549533 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58d5fa-58b5b7b7df-nzxwm" Oct 30 22:26:26.572440 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:26.572365 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58d5fa-58b5b7b7df-nzxwm" podStartSLOduration=4.074053649 podStartE2EDuration="4.572350818s" totalImagesPullingTime="498.297169ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:26.570903281 +0000 UTC m=+2752.741868462" watchObservedRunningTime="2026-10-30 22:26:26.572350818 +0000 UTC m=+2752.743315999" Oct 30 22:26:27.212426 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:27.210721 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:27.924796 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:27.922080 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:28.254736 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:28.252306 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:28.694704 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:28.694657 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:29.249992 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:29.249941 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:29.663413 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:29.660660 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:29.821425 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:29.815198 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:30.436402 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:30.436359 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:30.591422 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:30.585108 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:30.891632 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:30.891583 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:31.391072 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:31.391036 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5313f0-5d566fb777-cvvgv"] Oct 30 22:26:31.391322 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:31.391295 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5313f0-5d566fb777-cvvgv" podUID="08f68597-808f-47ed-9878-b8ce40d02287" containerName="maas-api" containerID="cri-o://cbc0d384215c4473acf87700fde11799fc3ae3f9ddfbb4be530460ba6ba6094d" gracePeriod=30 Oct 30 22:26:31.474723 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:31.474685 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:31.793117 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:31.786173 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:31.881730 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:31.881427 6720 generic.go:356] "Generic (PLEG): container finished" podID="08f68597-808f-47ed-9878-b8ce40d02287" containerID="cbc0d384215c4473acf87700fde11799fc3ae3f9ddfbb4be530460ba6ba6094d" exitCode=0 Oct 30 22:26:31.881730 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:31.881638 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5313f0-5d566fb777-cvvgv" event={"ID":"08f68597-808f-47ed-9878-b8ce40d02287","Type":"ContainerDied","Data":"cbc0d384215c4473acf87700fde11799fc3ae3f9ddfbb4be530460ba6ba6094d"} Oct 30 22:26:31.939399 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:31.939344 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:32.015847 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:32.015810 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5313f0-5d566fb777-cvvgv" Oct 30 22:26:32.069597 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:32.069521 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08f68597-808f-47ed-9878-b8ce40d02287-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08f68597-808f-47ed-9878-b8ce40d02287-openshift-service-ca\") pod \"08f68597-808f-47ed-9878-b8ce40d02287\" (UID: \"08f68597-808f-47ed-9878-b8ce40d02287\") " Oct 30 22:26:32.069597 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:32.069558 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08f68597-808f-47ed-9878-b8ce40d02287-kube-api-access-nhjjv\" (UniqueName: \"kubernetes.io/projected/08f68597-808f-47ed-9878-b8ce40d02287-kube-api-access-nhjjv\") pod \"08f68597-808f-47ed-9878-b8ce40d02287\" (UID: \"08f68597-808f-47ed-9878-b8ce40d02287\") " Oct 30 22:26:32.069894 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:32.069603 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08f68597-808f-47ed-9878-b8ce40d02287-metrics-certs\" (UniqueName: \"kubernetes.io/secret/08f68597-808f-47ed-9878-b8ce40d02287-metrics-certs\") pod \"08f68597-808f-47ed-9878-b8ce40d02287\" (UID: \"08f68597-808f-47ed-9878-b8ce40d02287\") " Oct 30 22:26:32.069894 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:32.069660 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08f68597-808f-47ed-9878-b8ce40d02287-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08f68597-808f-47ed-9878-b8ce40d02287-maas-api-tls\") pod \"08f68597-808f-47ed-9878-b8ce40d02287\" (UID: \"08f68597-808f-47ed-9878-b8ce40d02287\") " Oct 30 22:26:32.070050 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:32.069939 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08f68597-808f-47ed-9878-b8ce40d02287-openshift-service-ca" pod "08f68597-808f-47ed-9878-b8ce40d02287" (UID: "08f68597-808f-47ed-9878-b8ce40d02287"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:26:32.070336 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:32.070306 6720 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08f68597-808f-47ed-9878-b8ce40d02287-openshift-service-ca\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:32.072193 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:32.072163 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08f68597-808f-47ed-9878-b8ce40d02287-metrics-certs" pod "08f68597-808f-47ed-9878-b8ce40d02287" (UID: "08f68597-808f-47ed-9878-b8ce40d02287"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:26:32.072526 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:32.072507 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08f68597-808f-47ed-9878-b8ce40d02287-maas-api-tls" pod "08f68597-808f-47ed-9878-b8ce40d02287" (UID: "08f68597-808f-47ed-9878-b8ce40d02287"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:26:32.072526 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:32.072517 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08f68597-808f-47ed-9878-b8ce40d02287-kube-api-access-nhjjv" pod "08f68597-808f-47ed-9878-b8ce40d02287" (UID: "08f68597-808f-47ed-9878-b8ce40d02287"). InnerVolumeSpecName "kube-api-access-nhjjv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:26:32.086451 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:32.086425 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:32.171968 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:32.171934 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhjjv\" (UniqueName: \"kubernetes.io/projected/08f68597-808f-47ed-9878-b8ce40d02287-kube-api-access-nhjjv\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:32.171968 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:32.171966 6720 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08f68597-808f-47ed-9878-b8ce40d02287-metrics-certs\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:32.172215 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:32.171984 6720 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08f68597-808f-47ed-9878-b8ce40d02287-maas-api-tls\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:32.364355 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:32.364319 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:32.562767 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:32.562732 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58d5fa-58b5b7b7df-nzxwm" Oct 30 22:26:32.889840 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:32.889811 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5313f0-5d566fb777-cvvgv" Oct 30 22:26:32.890020 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:32.889814 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5313f0-5d566fb777-cvvgv" event={"ID":"08f68597-808f-47ed-9878-b8ce40d02287","Type":"ContainerDied","Data":"3d2094cd3f960a18ac70ca5fc53d78824becdfaaaf1f8b3f9bbd0535e357d1f1"} Oct 30 22:26:32.890020 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:32.889948 6720 scope.go:122] "RemoveContainer" containerID="cbc0d384215c4473acf87700fde11799fc3ae3f9ddfbb4be530460ba6ba6094d" Oct 30 22:26:32.915585 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:32.915556 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5313f0-5d566fb777-cvvgv"] Oct 30 22:26:32.919721 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:32.919692 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-5313f0-5d566fb777-cvvgv"] Oct 30 22:26:33.050412 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:33.041773 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:33.830084 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:33.830046 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:33.958295 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:33.958265 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08f68597-808f-47ed-9878-b8ce40d02287" path="/var/lib/kubelet/pods/08f68597-808f-47ed-9878-b8ce40d02287/volumes" Oct 30 22:26:34.559212 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:34.559182 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:34.931776 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:34.931744 6720 scope.go:122] "RemoveContainer" containerID="e93c564228de69eb6f83bd95c8127739133cf8f0e059e8e73f4e796ba48ad415" Oct 30 22:26:35.298597 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:35.298502 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:36.136625 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:36.136559 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:36.746767 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:36.746730 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:37.442505 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:37.442458 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:38.109258 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:38.109188 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:38.253326 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:38.253288 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:38.909185 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:38.909146 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:39.055228 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:39.055198 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:39.332513 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:39.332376 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:39.717707 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:39.717670 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:39.984587 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:39.984503 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-4b2bb0/test-model-4b2bb0-kserve-5df44c7bc5-mj9p7"] Oct 30 22:26:39.985588 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:39.985546 6720 cpu_manager.go:414] "containerMap: removing container" podUID="08f68597-808f-47ed-9878-b8ce40d02287" containerName="maas-api" Oct 30 22:26:39.985588 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:39.985574 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08f68597-808f-47ed-9878-b8ce40d02287" containerName="maas-api" Oct 30 22:26:39.985853 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:39.985784 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="08f68597-808f-47ed-9878-b8ce40d02287" containerName="maas-api" Oct 30 22:26:39.991993 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:39.991965 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4b2bb0/test-model-4b2bb0-kserve-5df44c7bc5-mj9p7" Oct 30 22:26:40.009769 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:40.005577 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-4b2bb0\"/\"test-model-4b2bb0-kserve-self-signed-certs\"" Oct 30 22:26:40.009769 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:40.005722 6720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-4b2bb0\"/\"openshift-service-ca.crt\"" Oct 30 22:26:40.009769 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:40.005764 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-4b2bb0\"/\"default-dockercfg-72hlf\"" Oct 30 22:26:40.011179 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:40.010611 6720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-4b2bb0\"/\"kube-root-ca.crt\"" Oct 30 22:26:40.013156 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:40.013126 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-4b2bb0/test-model-4b2bb0-kserve-5df44c7bc5-mj9p7"] Oct 30 22:26:40.019873 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:40.019851 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7f48858-397b-46fe-84b4-f19ef4afa227-dshm\") pod \"test-model-4b2bb0-kserve-5df44c7bc5-mj9p7\" (UID: \"e7f48858-397b-46fe-84b4-f19ef4afa227\") " pod="ai-tenant-e2e-inf-b-4b2bb0/test-model-4b2bb0-kserve-5df44c7bc5-mj9p7" Oct 30 22:26:40.020004 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:40.019914 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7f48858-397b-46fe-84b4-f19ef4afa227-model-cache\") pod \"test-model-4b2bb0-kserve-5df44c7bc5-mj9p7\" (UID: \"e7f48858-397b-46fe-84b4-f19ef4afa227\") " pod="ai-tenant-e2e-inf-b-4b2bb0/test-model-4b2bb0-kserve-5df44c7bc5-mj9p7" Oct 30 22:26:40.020004 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:40.019966 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7f48858-397b-46fe-84b4-f19ef4afa227-tmp-dir\") pod \"test-model-4b2bb0-kserve-5df44c7bc5-mj9p7\" (UID: \"e7f48858-397b-46fe-84b4-f19ef4afa227\") " pod="ai-tenant-e2e-inf-b-4b2bb0/test-model-4b2bb0-kserve-5df44c7bc5-mj9p7" Oct 30 22:26:40.020345 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:40.020051 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7f48858-397b-46fe-84b4-f19ef4afa227-tls-certs\") pod \"test-model-4b2bb0-kserve-5df44c7bc5-mj9p7\" (UID: \"e7f48858-397b-46fe-84b4-f19ef4afa227\") " pod="ai-tenant-e2e-inf-b-4b2bb0/test-model-4b2bb0-kserve-5df44c7bc5-mj9p7" Oct 30 22:26:40.020345 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:40.020085 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7f48858-397b-46fe-84b4-f19ef4afa227-home\") pod \"test-model-4b2bb0-kserve-5df44c7bc5-mj9p7\" (UID: \"e7f48858-397b-46fe-84b4-f19ef4afa227\") " pod="ai-tenant-e2e-inf-b-4b2bb0/test-model-4b2bb0-kserve-5df44c7bc5-mj9p7" Oct 30 22:26:40.020345 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:40.020196 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xpx6\" (UniqueName: \"kubernetes.io/projected/e7f48858-397b-46fe-84b4-f19ef4afa227-kube-api-access-2xpx6\") pod \"test-model-4b2bb0-kserve-5df44c7bc5-mj9p7\" (UID: \"e7f48858-397b-46fe-84b4-f19ef4afa227\") " pod="ai-tenant-e2e-inf-b-4b2bb0/test-model-4b2bb0-kserve-5df44c7bc5-mj9p7" Oct 30 22:26:40.104435 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:40.104405 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:40.121180 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:40.121147 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xpx6\" (UniqueName: \"kubernetes.io/projected/e7f48858-397b-46fe-84b4-f19ef4afa227-kube-api-access-2xpx6\") pod \"test-model-4b2bb0-kserve-5df44c7bc5-mj9p7\" (UID: \"e7f48858-397b-46fe-84b4-f19ef4afa227\") " pod="ai-tenant-e2e-inf-b-4b2bb0/test-model-4b2bb0-kserve-5df44c7bc5-mj9p7" Oct 30 22:26:40.121455 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:40.121434 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7f48858-397b-46fe-84b4-f19ef4afa227-dshm\") pod \"test-model-4b2bb0-kserve-5df44c7bc5-mj9p7\" (UID: \"e7f48858-397b-46fe-84b4-f19ef4afa227\") " pod="ai-tenant-e2e-inf-b-4b2bb0/test-model-4b2bb0-kserve-5df44c7bc5-mj9p7" Oct 30 22:26:40.121559 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:40.121512 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7f48858-397b-46fe-84b4-f19ef4afa227-model-cache\") pod \"test-model-4b2bb0-kserve-5df44c7bc5-mj9p7\" (UID: \"e7f48858-397b-46fe-84b4-f19ef4afa227\") " pod="ai-tenant-e2e-inf-b-4b2bb0/test-model-4b2bb0-kserve-5df44c7bc5-mj9p7" Oct 30 22:26:40.121628 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:40.121574 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7f48858-397b-46fe-84b4-f19ef4afa227-tmp-dir\") pod \"test-model-4b2bb0-kserve-5df44c7bc5-mj9p7\" (UID: \"e7f48858-397b-46fe-84b4-f19ef4afa227\") " pod="ai-tenant-e2e-inf-b-4b2bb0/test-model-4b2bb0-kserve-5df44c7bc5-mj9p7" Oct 30 22:26:40.121682 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:40.121663 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7f48858-397b-46fe-84b4-f19ef4afa227-tls-certs\") pod \"test-model-4b2bb0-kserve-5df44c7bc5-mj9p7\" (UID: \"e7f48858-397b-46fe-84b4-f19ef4afa227\") " pod="ai-tenant-e2e-inf-b-4b2bb0/test-model-4b2bb0-kserve-5df44c7bc5-mj9p7" Oct 30 22:26:40.121741 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:40.121727 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7f48858-397b-46fe-84b4-f19ef4afa227-home\") pod \"test-model-4b2bb0-kserve-5df44c7bc5-mj9p7\" (UID: \"e7f48858-397b-46fe-84b4-f19ef4afa227\") " pod="ai-tenant-e2e-inf-b-4b2bb0/test-model-4b2bb0-kserve-5df44c7bc5-mj9p7" Oct 30 22:26:40.122013 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:40.121989 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7f48858-397b-46fe-84b4-f19ef4afa227-model-cache\") pod \"test-model-4b2bb0-kserve-5df44c7bc5-mj9p7\" (UID: \"e7f48858-397b-46fe-84b4-f19ef4afa227\") " pod="ai-tenant-e2e-inf-b-4b2bb0/test-model-4b2bb0-kserve-5df44c7bc5-mj9p7" Oct 30 22:26:40.122313 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:40.122293 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7f48858-397b-46fe-84b4-f19ef4afa227-tmp-dir\") pod \"test-model-4b2bb0-kserve-5df44c7bc5-mj9p7\" (UID: \"e7f48858-397b-46fe-84b4-f19ef4afa227\") " pod="ai-tenant-e2e-inf-b-4b2bb0/test-model-4b2bb0-kserve-5df44c7bc5-mj9p7" Oct 30 22:26:40.122544 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:40.122523 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7f48858-397b-46fe-84b4-f19ef4afa227-home\") pod \"test-model-4b2bb0-kserve-5df44c7bc5-mj9p7\" (UID: \"e7f48858-397b-46fe-84b4-f19ef4afa227\") " pod="ai-tenant-e2e-inf-b-4b2bb0/test-model-4b2bb0-kserve-5df44c7bc5-mj9p7" Oct 30 22:26:40.124140 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:40.124118 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7f48858-397b-46fe-84b4-f19ef4afa227-dshm\") pod \"test-model-4b2bb0-kserve-5df44c7bc5-mj9p7\" (UID: \"e7f48858-397b-46fe-84b4-f19ef4afa227\") " pod="ai-tenant-e2e-inf-b-4b2bb0/test-model-4b2bb0-kserve-5df44c7bc5-mj9p7" Oct 30 22:26:40.124797 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:40.124774 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7f48858-397b-46fe-84b4-f19ef4afa227-tls-certs\") pod \"test-model-4b2bb0-kserve-5df44c7bc5-mj9p7\" (UID: \"e7f48858-397b-46fe-84b4-f19ef4afa227\") " pod="ai-tenant-e2e-inf-b-4b2bb0/test-model-4b2bb0-kserve-5df44c7bc5-mj9p7" Oct 30 22:26:40.136071 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:40.136044 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xpx6\" (UniqueName: \"kubernetes.io/projected/e7f48858-397b-46fe-84b4-f19ef4afa227-kube-api-access-2xpx6\") pod \"test-model-4b2bb0-kserve-5df44c7bc5-mj9p7\" (UID: \"e7f48858-397b-46fe-84b4-f19ef4afa227\") " pod="ai-tenant-e2e-inf-b-4b2bb0/test-model-4b2bb0-kserve-5df44c7bc5-mj9p7" Oct 30 22:26:40.332715 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:40.332621 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4b2bb0/test-model-4b2bb0-kserve-5df44c7bc5-mj9p7" Oct 30 22:26:40.443439 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:26:40.443401 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 22:26:40.687075 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:40.687049 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-4b2bb0/test-model-4b2bb0-kserve-5df44c7bc5-mj9p7"] Oct 30 22:26:40.688996 ip-10-0-137-181 kubenswrapper[6720]: W1030 22:26:40.688962 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode7f48858_397b_46fe_84b4_f19ef4afa227.slice/crio-61656d44f52ced7df7983a05a833ff51f651ad5b32d9e5960ad7689acd8ed94d WatchSource:0}: Error finding container 61656d44f52ced7df7983a05a833ff51f651ad5b32d9e5960ad7689acd8ed94d: Status 404 returned error can't find the container with id 61656d44f52ced7df7983a05a833ff51f651ad5b32d9e5960ad7689acd8ed94d Oct 30 22:26:40.722250 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:40.722210 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:40.957428 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:40.955417 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4b2bb0/test-model-4b2bb0-kserve-5df44c7bc5-mj9p7" event={"ID":"e7f48858-397b-46fe-84b4-f19ef4afa227","Type":"ContainerStarted","Data":"8365e8b876cd67268b16869213046f1b7ad67025e0be94df18bd17822f578524"} Oct 30 22:26:40.957428 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:40.955466 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4b2bb0/test-model-4b2bb0-kserve-5df44c7bc5-mj9p7" event={"ID":"e7f48858-397b-46fe-84b4-f19ef4afa227","Type":"ContainerStarted","Data":"61656d44f52ced7df7983a05a833ff51f651ad5b32d9e5960ad7689acd8ed94d"} Oct 30 22:26:40.978672 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:40.978614 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-4b2bb0/test-model-4b2bb0-kserve-5df44c7bc5-mj9p7" podStartSLOduration=1.978596702 podStartE2EDuration="1.978596702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:40.976868492 +0000 UTC m=+2767.147833674" watchObservedRunningTime="2026-10-30 22:26:40.978596702 +0000 UTC m=+2767.149561884" Oct 30 22:26:41.337544 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:41.337461 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-4b2bb0/test-model-4b2bb0-kserve-5df44c7bc5-mj9p7" Oct 30 22:26:41.644329 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:41.644295 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:42.380208 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:42.380176 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:42.535905 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:42.535867 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:42.676831 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:42.676739 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:43.184677 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:43.184642 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-df200a/test-model-df200a-kserve-7fc978d98f-jc487"] Oct 30 22:26:43.190681 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:43.190652 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-df200a/test-model-df200a-kserve-7fc978d98f-jc487" Oct 30 22:26:43.193795 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:43.193760 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-df200a/test-model-df200a-kserve-7fc978d98f-jc487"] Oct 30 22:26:43.209806 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:43.209766 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-df200a\"/\"default-dockercfg-85p7n\"" Oct 30 22:26:43.209969 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:43.209877 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-df200a\"/\"test-model-df200a-kserve-self-signed-certs\"" Oct 30 22:26:43.209969 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:43.209946 6720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-df200a\"/\"kube-root-ca.crt\"" Oct 30 22:26:43.210109 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:43.209999 6720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-df200a\"/\"openshift-service-ca.crt\"" Oct 30 22:26:43.281349 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:43.281305 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q98jl\" (UniqueName: \"kubernetes.io/projected/3360dc0f-497d-4211-a73f-c5ac0ff8f204-kube-api-access-q98jl\") pod \"test-model-df200a-kserve-7fc978d98f-jc487\" (UID: \"3360dc0f-497d-4211-a73f-c5ac0ff8f204\") " pod="ai-tenant-e2e-inf-a-df200a/test-model-df200a-kserve-7fc978d98f-jc487" Oct 30 22:26:43.281601 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:43.281555 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3360dc0f-497d-4211-a73f-c5ac0ff8f204-model-cache\") pod \"test-model-df200a-kserve-7fc978d98f-jc487\" (UID: \"3360dc0f-497d-4211-a73f-c5ac0ff8f204\") " pod="ai-tenant-e2e-inf-a-df200a/test-model-df200a-kserve-7fc978d98f-jc487" Oct 30 22:26:43.281804 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:43.281782 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3360dc0f-497d-4211-a73f-c5ac0ff8f204-tls-certs\") pod \"test-model-df200a-kserve-7fc978d98f-jc487\" (UID: \"3360dc0f-497d-4211-a73f-c5ac0ff8f204\") " pod="ai-tenant-e2e-inf-a-df200a/test-model-df200a-kserve-7fc978d98f-jc487" Oct 30 22:26:43.281863 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:43.281823 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3360dc0f-497d-4211-a73f-c5ac0ff8f204-dshm\") pod \"test-model-df200a-kserve-7fc978d98f-jc487\" (UID: \"3360dc0f-497d-4211-a73f-c5ac0ff8f204\") " pod="ai-tenant-e2e-inf-a-df200a/test-model-df200a-kserve-7fc978d98f-jc487" Oct 30 22:26:43.281917 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:43.281894 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3360dc0f-497d-4211-a73f-c5ac0ff8f204-tmp-dir\") pod \"test-model-df200a-kserve-7fc978d98f-jc487\" (UID: \"3360dc0f-497d-4211-a73f-c5ac0ff8f204\") " pod="ai-tenant-e2e-inf-a-df200a/test-model-df200a-kserve-7fc978d98f-jc487" Oct 30 22:26:43.282017 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:43.281999 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3360dc0f-497d-4211-a73f-c5ac0ff8f204-home\") pod \"test-model-df200a-kserve-7fc978d98f-jc487\" (UID: \"3360dc0f-497d-4211-a73f-c5ac0ff8f204\") " pod="ai-tenant-e2e-inf-a-df200a/test-model-df200a-kserve-7fc978d98f-jc487" Oct 30 22:26:43.295010 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:43.294976 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:43.388409 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:43.384108 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3360dc0f-497d-4211-a73f-c5ac0ff8f204-model-cache\") pod \"test-model-df200a-kserve-7fc978d98f-jc487\" (UID: \"3360dc0f-497d-4211-a73f-c5ac0ff8f204\") " pod="ai-tenant-e2e-inf-a-df200a/test-model-df200a-kserve-7fc978d98f-jc487" Oct 30 22:26:43.388409 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:43.384319 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3360dc0f-497d-4211-a73f-c5ac0ff8f204-tls-certs\") pod \"test-model-df200a-kserve-7fc978d98f-jc487\" (UID: \"3360dc0f-497d-4211-a73f-c5ac0ff8f204\") " pod="ai-tenant-e2e-inf-a-df200a/test-model-df200a-kserve-7fc978d98f-jc487" Oct 30 22:26:43.388409 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:43.384426 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3360dc0f-497d-4211-a73f-c5ac0ff8f204-dshm\") pod \"test-model-df200a-kserve-7fc978d98f-jc487\" (UID: \"3360dc0f-497d-4211-a73f-c5ac0ff8f204\") " pod="ai-tenant-e2e-inf-a-df200a/test-model-df200a-kserve-7fc978d98f-jc487" Oct 30 22:26:43.388409 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:43.384460 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3360dc0f-497d-4211-a73f-c5ac0ff8f204-tmp-dir\") pod \"test-model-df200a-kserve-7fc978d98f-jc487\" (UID: \"3360dc0f-497d-4211-a73f-c5ac0ff8f204\") " pod="ai-tenant-e2e-inf-a-df200a/test-model-df200a-kserve-7fc978d98f-jc487" Oct 30 22:26:43.388409 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:43.384569 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3360dc0f-497d-4211-a73f-c5ac0ff8f204-home\") pod \"test-model-df200a-kserve-7fc978d98f-jc487\" (UID: \"3360dc0f-497d-4211-a73f-c5ac0ff8f204\") " pod="ai-tenant-e2e-inf-a-df200a/test-model-df200a-kserve-7fc978d98f-jc487" Oct 30 22:26:43.388409 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:43.384620 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q98jl\" (UniqueName: \"kubernetes.io/projected/3360dc0f-497d-4211-a73f-c5ac0ff8f204-kube-api-access-q98jl\") pod \"test-model-df200a-kserve-7fc978d98f-jc487\" (UID: \"3360dc0f-497d-4211-a73f-c5ac0ff8f204\") " pod="ai-tenant-e2e-inf-a-df200a/test-model-df200a-kserve-7fc978d98f-jc487" Oct 30 22:26:43.388409 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:43.385607 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3360dc0f-497d-4211-a73f-c5ac0ff8f204-model-cache\") pod \"test-model-df200a-kserve-7fc978d98f-jc487\" (UID: \"3360dc0f-497d-4211-a73f-c5ac0ff8f204\") " pod="ai-tenant-e2e-inf-a-df200a/test-model-df200a-kserve-7fc978d98f-jc487" Oct 30 22:26:43.399437 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:43.396468 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3360dc0f-497d-4211-a73f-c5ac0ff8f204-tls-certs\") pod \"test-model-df200a-kserve-7fc978d98f-jc487\" (UID: \"3360dc0f-497d-4211-a73f-c5ac0ff8f204\") " pod="ai-tenant-e2e-inf-a-df200a/test-model-df200a-kserve-7fc978d98f-jc487" Oct 30 22:26:43.424474 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:43.403836 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3360dc0f-497d-4211-a73f-c5ac0ff8f204-tmp-dir\") pod \"test-model-df200a-kserve-7fc978d98f-jc487\" (UID: \"3360dc0f-497d-4211-a73f-c5ac0ff8f204\") " pod="ai-tenant-e2e-inf-a-df200a/test-model-df200a-kserve-7fc978d98f-jc487" Oct 30 22:26:43.424474 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:43.404090 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3360dc0f-497d-4211-a73f-c5ac0ff8f204-home\") pod \"test-model-df200a-kserve-7fc978d98f-jc487\" (UID: \"3360dc0f-497d-4211-a73f-c5ac0ff8f204\") " pod="ai-tenant-e2e-inf-a-df200a/test-model-df200a-kserve-7fc978d98f-jc487" Oct 30 22:26:43.424474 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:43.418855 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q98jl\" (UniqueName: \"kubernetes.io/projected/3360dc0f-497d-4211-a73f-c5ac0ff8f204-kube-api-access-q98jl\") pod \"test-model-df200a-kserve-7fc978d98f-jc487\" (UID: \"3360dc0f-497d-4211-a73f-c5ac0ff8f204\") " pod="ai-tenant-e2e-inf-a-df200a/test-model-df200a-kserve-7fc978d98f-jc487" Oct 30 22:26:43.425428 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:43.425400 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3360dc0f-497d-4211-a73f-c5ac0ff8f204-dshm\") pod \"test-model-df200a-kserve-7fc978d98f-jc487\" (UID: \"3360dc0f-497d-4211-a73f-c5ac0ff8f204\") " pod="ai-tenant-e2e-inf-a-df200a/test-model-df200a-kserve-7fc978d98f-jc487" Oct 30 22:26:43.513808 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:43.513720 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-df200a/test-model-df200a-kserve-7fc978d98f-jc487" Oct 30 22:26:43.608032 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:43.607852 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:43.870345 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:43.870251 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-df200a/test-model-df200a-kserve-7fc978d98f-jc487"] Oct 30 22:26:43.873752 ip-10-0-137-181 kubenswrapper[6720]: W1030 22:26:43.873704 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3360dc0f_497d_4211_a73f_c5ac0ff8f204.slice/crio-a73425fb5d0dadbbc42e1cfa9308d8a1af27f77a3c267da29e54405f9e48a200 WatchSource:0}: Error finding container a73425fb5d0dadbbc42e1cfa9308d8a1af27f77a3c267da29e54405f9e48a200: Status 404 returned error can't find the container with id a73425fb5d0dadbbc42e1cfa9308d8a1af27f77a3c267da29e54405f9e48a200 Oct 30 22:26:43.979829 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:43.979737 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-df200a/test-model-df200a-kserve-7fc978d98f-jc487" event={"ID":"3360dc0f-497d-4211-a73f-c5ac0ff8f204","Type":"ContainerStarted","Data":"b9cc50e667f7a065cc2c30ba3bc6625d8c292eb015208a9980eba973d6b9d88e"} Oct 30 22:26:43.979829 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:43.979800 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-df200a/test-model-df200a-kserve-7fc978d98f-jc487" event={"ID":"3360dc0f-497d-4211-a73f-c5ac0ff8f204","Type":"ContainerStarted","Data":"a73425fb5d0dadbbc42e1cfa9308d8a1af27f77a3c267da29e54405f9e48a200"} Oct 30 22:26:44.003770 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:44.003674 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-df200a/test-model-df200a-kserve-7fc978d98f-jc487" podStartSLOduration=1.003652994 podStartE2EDuration="1.003652994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:43.999712978 +0000 UTC m=+2770.170678203" watchObservedRunningTime="2026-10-30 22:26:44.003652994 +0000 UTC m=+2770.174618180" Oct 30 22:26:44.357447 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:44.356049 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:44.514547 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:44.514516 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-df200a/test-model-df200a-kserve-7fc978d98f-jc487" Oct 30 22:26:45.066462 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:45.066427 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:45.223596 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:45.223561 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:46.038410 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:46.032104 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:47.000234 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:47.000182 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:47.825198 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:47.824796 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:48.622102 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:48.622046 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:49.372726 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:49.372329 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:50.030775 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:50.030744 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:50.202607 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:50.202576 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:50.334996 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:50.334915 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-4b2bb0/test-model-4b2bb0-kserve-5df44c7bc5-mj9p7" Oct 30 22:26:50.350980 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:50.350948 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-4b2bb0/test-model-4b2bb0-kserve-5df44c7bc5-mj9p7" Oct 30 22:26:51.087008 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:51.086978 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-4b2bb0/test-model-4b2bb0-kserve-5df44c7bc5-mj9p7" Oct 30 22:26:51.235133 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:51.235084 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:52.095913 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:52.095877 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:52.943412 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:52.943359 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:53.513987 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:53.513946 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-df200a/test-model-df200a-kserve-7fc978d98f-jc487" Oct 30 22:26:53.527325 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:53.527292 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-df200a/test-model-df200a-kserve-7fc978d98f-jc487" Oct 30 22:26:53.869247 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:53.869180 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:54.037418 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:54.028788 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:54.149378 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:54.149288 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-df200a/test-model-df200a-kserve-7fc978d98f-jc487" Oct 30 22:26:54.647342 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:54.647305 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:55.339446 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:55.339372 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:55.524413 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:55.524255 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:55.687766 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:55.687694 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:55.842570 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:55.842521 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:56.139974 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:56.139929 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:56.800140 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:56.800089 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:57.511117 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:57.511077 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:57.673116 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:57.673085 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:57.945960 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:57.945917 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-v4kbr"] Oct 30 22:26:57.952245 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:57.952209 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-v4kbr" Oct 30 22:26:57.969578 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:57.969552 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-v4kbr"] Oct 30 22:26:58.083927 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:58.083886 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/744e223f-d9d1-45c3-9f37-b4d29091852c-istiod-ca-cert\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-v4kbr\" (UID: \"744e223f-d9d1-45c3-9f37-b4d29091852c\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-v4kbr" Oct 30 22:26:58.084140 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:58.083967 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/744e223f-d9d1-45c3-9f37-b4d29091852c-istio-token\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-v4kbr\" (UID: \"744e223f-d9d1-45c3-9f37-b4d29091852c\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-v4kbr" Oct 30 22:26:58.084140 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:58.083986 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/744e223f-d9d1-45c3-9f37-b4d29091852c-istio-envoy\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-v4kbr\" (UID: \"744e223f-d9d1-45c3-9f37-b4d29091852c\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-v4kbr" Oct 30 22:26:58.084140 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:58.084009 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/744e223f-d9d1-45c3-9f37-b4d29091852c-istio-data\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-v4kbr\" (UID: \"744e223f-d9d1-45c3-9f37-b4d29091852c\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-v4kbr" Oct 30 22:26:58.084140 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:58.084070 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/744e223f-d9d1-45c3-9f37-b4d29091852c-workload-socket\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-v4kbr\" (UID: \"744e223f-d9d1-45c3-9f37-b4d29091852c\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-v4kbr" Oct 30 22:26:58.084364 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:58.084304 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/744e223f-d9d1-45c3-9f37-b4d29091852c-credential-socket\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-v4kbr\" (UID: \"744e223f-d9d1-45c3-9f37-b4d29091852c\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-v4kbr" Oct 30 22:26:58.084364 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:58.084349 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnk7n\" (UniqueName: \"kubernetes.io/projected/744e223f-d9d1-45c3-9f37-b4d29091852c-kube-api-access-wnk7n\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-v4kbr\" (UID: \"744e223f-d9d1-45c3-9f37-b4d29091852c\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-v4kbr" Oct 30 22:26:58.084491 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:58.084380 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/744e223f-d9d1-45c3-9f37-b4d29091852c-istio-podinfo\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-v4kbr\" (UID: \"744e223f-d9d1-45c3-9f37-b4d29091852c\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-v4kbr" Oct 30 22:26:58.084611 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:58.084595 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/744e223f-d9d1-45c3-9f37-b4d29091852c-workload-certs\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-v4kbr\" (UID: \"744e223f-d9d1-45c3-9f37-b4d29091852c\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-v4kbr" Oct 30 22:26:58.186164 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:58.186132 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/744e223f-d9d1-45c3-9f37-b4d29091852c-workload-certs\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-v4kbr\" (UID: \"744e223f-d9d1-45c3-9f37-b4d29091852c\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-v4kbr" Oct 30 22:26:58.186343 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:58.186268 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/744e223f-d9d1-45c3-9f37-b4d29091852c-istiod-ca-cert\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-v4kbr\" (UID: \"744e223f-d9d1-45c3-9f37-b4d29091852c\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-v4kbr" Oct 30 22:26:58.186343 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:58.186311 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/744e223f-d9d1-45c3-9f37-b4d29091852c-istio-token\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-v4kbr\" (UID: \"744e223f-d9d1-45c3-9f37-b4d29091852c\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-v4kbr" Oct 30 22:26:58.186343 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:58.186333 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/744e223f-d9d1-45c3-9f37-b4d29091852c-istio-envoy\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-v4kbr\" (UID: \"744e223f-d9d1-45c3-9f37-b4d29091852c\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-v4kbr" Oct 30 22:26:58.186536 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:58.186364 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/744e223f-d9d1-45c3-9f37-b4d29091852c-istio-data\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-v4kbr\" (UID: \"744e223f-d9d1-45c3-9f37-b4d29091852c\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-v4kbr" Oct 30 22:26:58.186536 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:58.186435 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/744e223f-d9d1-45c3-9f37-b4d29091852c-workload-socket\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-v4kbr\" (UID: \"744e223f-d9d1-45c3-9f37-b4d29091852c\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-v4kbr" Oct 30 22:26:58.186536 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:58.186476 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/744e223f-d9d1-45c3-9f37-b4d29091852c-credential-socket\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-v4kbr\" (UID: \"744e223f-d9d1-45c3-9f37-b4d29091852c\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-v4kbr" Oct 30 22:26:58.186536 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:58.186502 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/744e223f-d9d1-45c3-9f37-b4d29091852c-workload-certs\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-v4kbr\" (UID: \"744e223f-d9d1-45c3-9f37-b4d29091852c\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-v4kbr" Oct 30 22:26:58.186765 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:58.186555 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnk7n\" (UniqueName: \"kubernetes.io/projected/744e223f-d9d1-45c3-9f37-b4d29091852c-kube-api-access-wnk7n\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-v4kbr\" (UID: \"744e223f-d9d1-45c3-9f37-b4d29091852c\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-v4kbr" Oct 30 22:26:58.186765 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:58.186604 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/744e223f-d9d1-45c3-9f37-b4d29091852c-istio-podinfo\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-v4kbr\" (UID: \"744e223f-d9d1-45c3-9f37-b4d29091852c\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-v4kbr" Oct 30 22:26:58.187052 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:58.187025 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/744e223f-d9d1-45c3-9f37-b4d29091852c-credential-socket\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-v4kbr\" (UID: \"744e223f-d9d1-45c3-9f37-b4d29091852c\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-v4kbr" Oct 30 22:26:58.187052 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:58.187049 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/744e223f-d9d1-45c3-9f37-b4d29091852c-workload-socket\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-v4kbr\" (UID: \"744e223f-d9d1-45c3-9f37-b4d29091852c\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-v4kbr" Oct 30 22:26:58.187209 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:58.187100 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/744e223f-d9d1-45c3-9f37-b4d29091852c-istio-data\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-v4kbr\" (UID: \"744e223f-d9d1-45c3-9f37-b4d29091852c\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-v4kbr" Oct 30 22:26:58.187299 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:58.187143 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/744e223f-d9d1-45c3-9f37-b4d29091852c-istiod-ca-cert\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-v4kbr\" (UID: \"744e223f-d9d1-45c3-9f37-b4d29091852c\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-v4kbr" Oct 30 22:26:58.188904 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:58.188881 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/744e223f-d9d1-45c3-9f37-b4d29091852c-istio-envoy\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-v4kbr\" (UID: \"744e223f-d9d1-45c3-9f37-b4d29091852c\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-v4kbr" Oct 30 22:26:58.189320 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:58.189303 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/744e223f-d9d1-45c3-9f37-b4d29091852c-istio-podinfo\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-v4kbr\" (UID: \"744e223f-d9d1-45c3-9f37-b4d29091852c\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-v4kbr" Oct 30 22:26:58.200506 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:58.200448 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/744e223f-d9d1-45c3-9f37-b4d29091852c-istio-token\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-v4kbr\" (UID: \"744e223f-d9d1-45c3-9f37-b4d29091852c\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-v4kbr" Oct 30 22:26:58.204237 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:58.204210 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnk7n\" (UniqueName: \"kubernetes.io/projected/744e223f-d9d1-45c3-9f37-b4d29091852c-kube-api-access-wnk7n\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-v4kbr\" (UID: \"744e223f-d9d1-45c3-9f37-b4d29091852c\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-v4kbr" Oct 30 22:26:58.275494 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:58.275457 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-v4kbr" Oct 30 22:26:58.340689 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:58.340653 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:26:58.632795 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:58.632761 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-v4kbr"] Oct 30 22:26:58.633409 ip-10-0-137-181 kubenswrapper[6720]: W1030 22:26:58.633358 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod744e223f_d9d1_45c3_9f37_b4d29091852c.slice/crio-0bd8296e97511290b70a3112c62686d2a003628a96bec68b8b0d2766cda04a13 WatchSource:0}: Error finding container 0bd8296e97511290b70a3112c62686d2a003628a96bec68b8b0d2766cda04a13: Status 404 returned error can't find the container with id 0bd8296e97511290b70a3112c62686d2a003628a96bec68b8b0d2766cda04a13 Oct 30 22:26:58.636533 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:58.636503 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:26:58.636602 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:58.636568 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:26:58.636644 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:58.636624 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:26:59.169937 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:59.169904 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-v4kbr" event={"ID":"744e223f-d9d1-45c3-9f37-b4d29091852c","Type":"ContainerStarted","Data":"bedcecf53043ca5387632a0ffd82fb1bb5179d55f20c8e358e8bf667d5e3237d"} Oct 30 22:26:59.170485 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:59.169955 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-v4kbr" event={"ID":"744e223f-d9d1-45c3-9f37-b4d29091852c","Type":"ContainerStarted","Data":"0bd8296e97511290b70a3112c62686d2a003628a96bec68b8b0d2766cda04a13"} Oct 30 22:26:59.208732 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:59.208646 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-v4kbr" podStartSLOduration=2.208626724 podStartE2EDuration="2.208626724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:59.206807963 +0000 UTC m=+2785.377773177" watchObservedRunningTime="2026-10-30 22:26:59.208626724 +0000 UTC m=+2785.379591905" Oct 30 22:26:59.275970 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:59.275932 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-v4kbr" Oct 30 22:26:59.278013 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:59.277985 6720 patch_prober.go:29] interesting pod/e2e-inf-a-120cb1-openshift-default-d97c644bf-v4kbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 22:26:59.278165 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:59.278034 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-v4kbr" podUID="744e223f-d9d1-45c3-9f37-b4d29091852c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 22:26:59.368588 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:26:59.368547 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:00.134410 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:00.133006 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:00.275928 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:00.275891 6720 patch_prober.go:29] interesting pod/e2e-inf-a-120cb1-openshift-default-d97c644bf-v4kbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 22:27:00.276359 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:00.275950 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-v4kbr" podUID="744e223f-d9d1-45c3-9f37-b4d29091852c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 22:27:00.974020 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:00.973981 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:01.276806 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:01.276722 6720 patch_prober.go:29] interesting pod/e2e-inf-a-120cb1-openshift-default-d97c644bf-v4kbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 22:27:01.276806 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:01.276792 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-v4kbr" podUID="744e223f-d9d1-45c3-9f37-b4d29091852c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 22:27:01.820436 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:01.813055 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:02.276344 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:02.276314 6720 patch_prober.go:29] interesting pod/e2e-inf-a-120cb1-openshift-default-d97c644bf-v4kbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 22:27:02.276599 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:02.276366 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-v4kbr" podUID="744e223f-d9d1-45c3-9f37-b4d29091852c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 22:27:02.758631 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:02.758578 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:03.276112 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:03.276080 6720 patch_prober.go:29] interesting pod/e2e-inf-a-120cb1-openshift-default-d97c644bf-v4kbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 22:27:03.276277 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:03.276135 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-v4kbr" podUID="744e223f-d9d1-45c3-9f37-b4d29091852c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 22:27:03.698241 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:03.698207 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:03.989243 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:03.989155 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:04.058028 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:04.054861 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-58d5fa/test-model-58d5fa-kserve-746d5495f7-88wps"] Oct 30 22:27:04.079070 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:04.079021 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-58d5fa/test-model-58d5fa-kserve-746d5495f7-88wps" Oct 30 22:27:04.118298 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:04.118268 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-58d5fa\"/\"test-model-58d5fa-kserve-self-signed-certs\"" Oct 30 22:27:04.122821 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:04.119870 6720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-58d5fa\"/\"kube-root-ca.crt\"" Oct 30 22:27:04.122821 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:04.119958 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-58d5fa\"/\"default-dockercfg-bx9d8\"" Oct 30 22:27:04.122821 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:04.121114 6720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-58d5fa\"/\"openshift-service-ca.crt\"" Oct 30 22:27:04.127196 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:04.127169 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-58d5fa/test-model-58d5fa-kserve-746d5495f7-88wps"] Oct 30 22:27:04.128494 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:04.128467 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ae33413-cc17-4bbb-ac91-ec406f02c125-dshm\") pod \"test-model-58d5fa-kserve-746d5495f7-88wps\" (UID: \"5ae33413-cc17-4bbb-ac91-ec406f02c125\") " pod="ai-tenant-e2e-inf-b-58d5fa/test-model-58d5fa-kserve-746d5495f7-88wps" Oct 30 22:27:04.128718 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:04.128681 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ae33413-cc17-4bbb-ac91-ec406f02c125-tls-certs\") pod \"test-model-58d5fa-kserve-746d5495f7-88wps\" (UID: \"5ae33413-cc17-4bbb-ac91-ec406f02c125\") " pod="ai-tenant-e2e-inf-b-58d5fa/test-model-58d5fa-kserve-746d5495f7-88wps" Oct 30 22:27:04.128977 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:04.128947 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ae33413-cc17-4bbb-ac91-ec406f02c125-model-cache\") pod \"test-model-58d5fa-kserve-746d5495f7-88wps\" (UID: \"5ae33413-cc17-4bbb-ac91-ec406f02c125\") " pod="ai-tenant-e2e-inf-b-58d5fa/test-model-58d5fa-kserve-746d5495f7-88wps" Oct 30 22:27:04.129269 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:04.129253 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ae33413-cc17-4bbb-ac91-ec406f02c125-home\") pod \"test-model-58d5fa-kserve-746d5495f7-88wps\" (UID: \"5ae33413-cc17-4bbb-ac91-ec406f02c125\") " pod="ai-tenant-e2e-inf-b-58d5fa/test-model-58d5fa-kserve-746d5495f7-88wps" Oct 30 22:27:04.129472 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:04.129425 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdrtw\" (UniqueName: \"kubernetes.io/projected/5ae33413-cc17-4bbb-ac91-ec406f02c125-kube-api-access-vdrtw\") pod \"test-model-58d5fa-kserve-746d5495f7-88wps\" (UID: \"5ae33413-cc17-4bbb-ac91-ec406f02c125\") " pod="ai-tenant-e2e-inf-b-58d5fa/test-model-58d5fa-kserve-746d5495f7-88wps" Oct 30 22:27:04.130852 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:04.130832 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ae33413-cc17-4bbb-ac91-ec406f02c125-tmp-dir\") pod \"test-model-58d5fa-kserve-746d5495f7-88wps\" (UID: \"5ae33413-cc17-4bbb-ac91-ec406f02c125\") " pod="ai-tenant-e2e-inf-b-58d5fa/test-model-58d5fa-kserve-746d5495f7-88wps" Oct 30 22:27:04.183367 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:04.183322 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:04.237590 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:04.237549 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ae33413-cc17-4bbb-ac91-ec406f02c125-home\") pod \"test-model-58d5fa-kserve-746d5495f7-88wps\" (UID: \"5ae33413-cc17-4bbb-ac91-ec406f02c125\") " pod="ai-tenant-e2e-inf-b-58d5fa/test-model-58d5fa-kserve-746d5495f7-88wps" Oct 30 22:27:04.237853 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:04.237833 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdrtw\" (UniqueName: \"kubernetes.io/projected/5ae33413-cc17-4bbb-ac91-ec406f02c125-kube-api-access-vdrtw\") pod \"test-model-58d5fa-kserve-746d5495f7-88wps\" (UID: \"5ae33413-cc17-4bbb-ac91-ec406f02c125\") " pod="ai-tenant-e2e-inf-b-58d5fa/test-model-58d5fa-kserve-746d5495f7-88wps" Oct 30 22:27:04.238063 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:04.238048 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ae33413-cc17-4bbb-ac91-ec406f02c125-tmp-dir\") pod \"test-model-58d5fa-kserve-746d5495f7-88wps\" (UID: \"5ae33413-cc17-4bbb-ac91-ec406f02c125\") " pod="ai-tenant-e2e-inf-b-58d5fa/test-model-58d5fa-kserve-746d5495f7-88wps" Oct 30 22:27:04.238242 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:04.238228 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ae33413-cc17-4bbb-ac91-ec406f02c125-dshm\") pod \"test-model-58d5fa-kserve-746d5495f7-88wps\" (UID: \"5ae33413-cc17-4bbb-ac91-ec406f02c125\") " pod="ai-tenant-e2e-inf-b-58d5fa/test-model-58d5fa-kserve-746d5495f7-88wps" Oct 30 22:27:04.238371 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:04.238357 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ae33413-cc17-4bbb-ac91-ec406f02c125-tls-certs\") pod \"test-model-58d5fa-kserve-746d5495f7-88wps\" (UID: \"5ae33413-cc17-4bbb-ac91-ec406f02c125\") " pod="ai-tenant-e2e-inf-b-58d5fa/test-model-58d5fa-kserve-746d5495f7-88wps" Oct 30 22:27:04.238690 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:04.238668 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ae33413-cc17-4bbb-ac91-ec406f02c125-model-cache\") pod \"test-model-58d5fa-kserve-746d5495f7-88wps\" (UID: \"5ae33413-cc17-4bbb-ac91-ec406f02c125\") " pod="ai-tenant-e2e-inf-b-58d5fa/test-model-58d5fa-kserve-746d5495f7-88wps" Oct 30 22:27:04.239408 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:04.239331 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ae33413-cc17-4bbb-ac91-ec406f02c125-model-cache\") pod \"test-model-58d5fa-kserve-746d5495f7-88wps\" (UID: \"5ae33413-cc17-4bbb-ac91-ec406f02c125\") " pod="ai-tenant-e2e-inf-b-58d5fa/test-model-58d5fa-kserve-746d5495f7-88wps" Oct 30 22:27:04.287225 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:04.287187 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ae33413-cc17-4bbb-ac91-ec406f02c125-tmp-dir\") pod \"test-model-58d5fa-kserve-746d5495f7-88wps\" (UID: \"5ae33413-cc17-4bbb-ac91-ec406f02c125\") " pod="ai-tenant-e2e-inf-b-58d5fa/test-model-58d5fa-kserve-746d5495f7-88wps" Oct 30 22:27:04.287989 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:04.287964 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ae33413-cc17-4bbb-ac91-ec406f02c125-home\") pod \"test-model-58d5fa-kserve-746d5495f7-88wps\" (UID: \"5ae33413-cc17-4bbb-ac91-ec406f02c125\") " pod="ai-tenant-e2e-inf-b-58d5fa/test-model-58d5fa-kserve-746d5495f7-88wps" Oct 30 22:27:04.323418 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:04.309707 6720 patch_prober.go:29] interesting pod/e2e-inf-a-120cb1-openshift-default-d97c644bf-v4kbr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 22:27:04.323418 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:04.309765 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-v4kbr" podUID="744e223f-d9d1-45c3-9f37-b4d29091852c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 22:27:04.396413 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:04.353513 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdrtw\" (UniqueName: \"kubernetes.io/projected/5ae33413-cc17-4bbb-ac91-ec406f02c125-kube-api-access-vdrtw\") pod \"test-model-58d5fa-kserve-746d5495f7-88wps\" (UID: \"5ae33413-cc17-4bbb-ac91-ec406f02c125\") " pod="ai-tenant-e2e-inf-b-58d5fa/test-model-58d5fa-kserve-746d5495f7-88wps" Oct 30 22:27:04.418101 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:04.418062 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ae33413-cc17-4bbb-ac91-ec406f02c125-tls-certs\") pod \"test-model-58d5fa-kserve-746d5495f7-88wps\" (UID: \"5ae33413-cc17-4bbb-ac91-ec406f02c125\") " pod="ai-tenant-e2e-inf-b-58d5fa/test-model-58d5fa-kserve-746d5495f7-88wps" Oct 30 22:27:04.431288 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:04.431252 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ae33413-cc17-4bbb-ac91-ec406f02c125-dshm\") pod \"test-model-58d5fa-kserve-746d5495f7-88wps\" (UID: \"5ae33413-cc17-4bbb-ac91-ec406f02c125\") " pod="ai-tenant-e2e-inf-b-58d5fa/test-model-58d5fa-kserve-746d5495f7-88wps" Oct 30 22:27:04.464777 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:04.464746 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-58d5fa/test-model-58d5fa-kserve-746d5495f7-88wps" Oct 30 22:27:04.706839 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:04.706786 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:04.892739 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:04.892708 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-58d5fa/test-model-58d5fa-kserve-746d5495f7-88wps"] Oct 30 22:27:04.894618 ip-10-0-137-181 kubenswrapper[6720]: W1030 22:27:04.894568 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5ae33413_cc17_4bbb_ac91_ec406f02c125.slice/crio-93529e161250d94e3e9c5b652f164e8b69545d942bbdc85a5c9c4d2e18414e5f WatchSource:0}: Error finding container 93529e161250d94e3e9c5b652f164e8b69545d942bbdc85a5c9c4d2e18414e5f: Status 404 returned error can't find the container with id 93529e161250d94e3e9c5b652f164e8b69545d942bbdc85a5c9c4d2e18414e5f Oct 30 22:27:05.330462 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:05.330359 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-v4kbr" Oct 30 22:27:05.331281 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:05.331256 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-v4kbr" Oct 30 22:27:05.333535 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:05.333515 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-v4kbr" Oct 30 22:27:05.376327 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:05.376292 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-58d5fa/test-model-58d5fa-kserve-746d5495f7-88wps" event={"ID":"5ae33413-cc17-4bbb-ac91-ec406f02c125","Type":"ContainerStarted","Data":"2fe82e1a7c0442818f6a40b742114d7597fc7ea14cb4303861fb5ea1e5d79bc3"} Oct 30 22:27:05.376543 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:05.376337 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-58d5fa/test-model-58d5fa-kserve-746d5495f7-88wps" event={"ID":"5ae33413-cc17-4bbb-ac91-ec406f02c125","Type":"ContainerStarted","Data":"93529e161250d94e3e9c5b652f164e8b69545d942bbdc85a5c9c4d2e18414e5f"} Oct 30 22:27:05.445722 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:05.445668 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-58d5fa/test-model-58d5fa-kserve-746d5495f7-88wps" podStartSLOduration=2.445650516 podStartE2EDuration="2.445650516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:05.44374779 +0000 UTC m=+2791.614712975" watchObservedRunningTime="2026-10-30 22:27:05.445650516 +0000 UTC m=+2791.616615744" Oct 30 22:27:05.466143 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:05.466107 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-58d5fa/test-model-58d5fa-kserve-746d5495f7-88wps" Oct 30 22:27:05.582797 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:05.582275 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:06.434411 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:06.434017 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:07.251269 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:07.249098 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:08.152814 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:08.152777 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:08.803421 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:08.802574 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:09.477798 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:09.477762 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:09.700921 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:09.700879 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:10.256952 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:10.256876 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:10.405209 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:27:10.405166 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 22:27:10.886430 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:10.886335 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:11.531321 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:11.531283 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:12.120680 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:12.120635 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:12.918553 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:12.918416 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:13.686004 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:13.685863 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:14.341001 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:14.340964 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:14.465833 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:14.465803 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-58d5fa/test-model-58d5fa-kserve-746d5495f7-88wps" Oct 30 22:27:14.479340 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:14.479314 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-58d5fa/test-model-58d5fa-kserve-746d5495f7-88wps" Oct 30 22:27:14.975411 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:14.970434 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:15.173199 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:15.173166 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:15.318977 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:15.318871 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:15.500258 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:15.500228 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-58d5fa/test-model-58d5fa-kserve-746d5495f7-88wps" Oct 30 22:27:15.939708 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:15.939621 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:16.605230 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:16.605181 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:17.248200 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:17.248150 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:18.124995 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:18.119007 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:18.799164 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:18.799118 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:19.433228 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:19.433009 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:20.137987 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:20.137946 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:20.838571 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:20.838518 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:21.452411 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:21.445539 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:22.156557 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:22.156525 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:22.828084 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:22.828037 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:23.450278 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:23.438578 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:24.210408 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:24.203554 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:24.827364 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:24.827329 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:25.643409 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:25.640212 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:26.545300 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:26.545253 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:27.416583 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:27.416550 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:28.615892 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:28.615860 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:29.166405 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:29.166350 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:29.310535 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:29.310486 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:29.456373 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:29.456287 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:29.967236 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:29.967190 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:30.253815 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:30.253734 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:30.397328 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:30.397271 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:31.063442 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:31.063264 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:31.912952 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:31.912912 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:32.742938 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:32.742897 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:33.528713 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:33.528676 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:34.163044 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:34.162991 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:34.916432 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:34.916321 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:35.681617 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:35.678297 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:36.306604 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:36.306566 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:37.033550 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:37.033510 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:37.912414 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:37.911023 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:38.510884 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:38.510848 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:39.178505 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:39.169678 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:39.929078 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:39.929018 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:40.404614 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:27:40.404575 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 22:27:40.535410 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:40.529913 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:41.329737 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:41.329701 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:42.036842 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:42.034867 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:42.617687 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:42.617626 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:43.359242 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:43.359204 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:44.091216 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:44.091179 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:44.809418 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:44.798574 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:45.529422 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:45.529150 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:46.238259 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:46.238224 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:46.411136 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:46.410083 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:46.652415 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:46.637632 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:46.882226 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:46.882197 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:47.401161 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:47.401084 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:48.127110 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:48.126929 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:48.791322 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:48.791287 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:48.947072 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:48.947034 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:49.567876 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:49.567837 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:50.164295 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:50.164247 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:50.323236 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:50.323194 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:50.610993 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:50.610962 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:50.763872 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:50.763828 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:51.132299 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:51.132261 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:51.424526 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:51.422016 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:52.299026 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:52.298994 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:53.016951 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:53.016910 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:53.810472 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:53.810434 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:54.447413 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:54.443376 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:55.180000 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:55.179969 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:55.330887 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:55.330858 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:56.011046 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:56.011004 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:56.443685 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:56.443650 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:56.593164 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:56.593129 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:57.102939 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:57.102913 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:57.252005 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:57.251962 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:57.436979 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:57.436884 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:57.615999 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:57.615957 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:57.922959 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:57.922918 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:58.559936 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:58.559901 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:58.708090 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:58.708059 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:58.832046 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:58.831958 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-7v8vz"] Oct 30 22:27:58.832975 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:58.832636 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-7v8vz" podUID="85e8605a-0cd7-4eaf-84e2-99e19e9ebebc" containerName="istio-proxy" containerID="cri-o://81ddd0d3b4e6e80f74ae75ff158a5885e9abe291aee626002650b947fe949ccf" gracePeriod=30 Oct 30 22:27:58.877537 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:58.877502 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:59.049410 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:59.047983 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:27:59.672059 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:27:59.672023 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:00.342408 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:00.341899 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:01.121497 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:01.120868 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:01.840194 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:01.835134 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:02.051243 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:02.051160 6720 patch_prober.go:29] interesting pod/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-7v8vz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:28:02.051243 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:02.051218 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-7v8vz" podUID="85e8605a-0cd7-4eaf-84e2-99e19e9ebebc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:28:02.527559 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:02.527499 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:03.223290 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:03.223246 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:03.866104 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:03.865403 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:03.980274 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:03.980238 6720 generic.go:356] "Generic (PLEG): container finished" podID="85e8605a-0cd7-4eaf-84e2-99e19e9ebebc" containerID="81ddd0d3b4e6e80f74ae75ff158a5885e9abe291aee626002650b947fe949ccf" exitCode=0 Oct 30 22:28:03.980274 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:03.980278 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-7v8vz" event={"ID":"85e8605a-0cd7-4eaf-84e2-99e19e9ebebc","Type":"ContainerDied","Data":"81ddd0d3b4e6e80f74ae75ff158a5885e9abe291aee626002650b947fe949ccf"} Oct 30 22:28:04.038041 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:04.037999 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:04.157715 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:04.157690 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-7v8vz" Oct 30 22:28:04.176895 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:04.176864 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:04.197833 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:04.197802 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85e8605a-0cd7-4eaf-84e2-99e19e9ebebc-istio-token\" (UniqueName: \"kubernetes.io/projected/85e8605a-0cd7-4eaf-84e2-99e19e9ebebc-istio-token\") pod \"85e8605a-0cd7-4eaf-84e2-99e19e9ebebc\" (UID: \"85e8605a-0cd7-4eaf-84e2-99e19e9ebebc\") " Oct 30 22:28:04.198016 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:04.197878 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85e8605a-0cd7-4eaf-84e2-99e19e9ebebc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85e8605a-0cd7-4eaf-84e2-99e19e9ebebc-istiod-ca-cert\") pod \"85e8605a-0cd7-4eaf-84e2-99e19e9ebebc\" (UID: \"85e8605a-0cd7-4eaf-84e2-99e19e9ebebc\") " Oct 30 22:28:04.198016 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:04.197912 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85e8605a-0cd7-4eaf-84e2-99e19e9ebebc-kube-api-access-74wvk\" (UniqueName: \"kubernetes.io/projected/85e8605a-0cd7-4eaf-84e2-99e19e9ebebc-kube-api-access-74wvk\") pod \"85e8605a-0cd7-4eaf-84e2-99e19e9ebebc\" (UID: \"85e8605a-0cd7-4eaf-84e2-99e19e9ebebc\") " Oct 30 22:28:04.198016 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:04.198004 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85e8605a-0cd7-4eaf-84e2-99e19e9ebebc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85e8605a-0cd7-4eaf-84e2-99e19e9ebebc-istio-data\") pod \"85e8605a-0cd7-4eaf-84e2-99e19e9ebebc\" (UID: \"85e8605a-0cd7-4eaf-84e2-99e19e9ebebc\") " Oct 30 22:28:04.198185 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:04.198046 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85e8605a-0cd7-4eaf-84e2-99e19e9ebebc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85e8605a-0cd7-4eaf-84e2-99e19e9ebebc-istio-envoy\") pod \"85e8605a-0cd7-4eaf-84e2-99e19e9ebebc\" (UID: \"85e8605a-0cd7-4eaf-84e2-99e19e9ebebc\") " Oct 30 22:28:04.198185 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:04.198089 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85e8605a-0cd7-4eaf-84e2-99e19e9ebebc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85e8605a-0cd7-4eaf-84e2-99e19e9ebebc-credential-socket\") pod \"85e8605a-0cd7-4eaf-84e2-99e19e9ebebc\" (UID: \"85e8605a-0cd7-4eaf-84e2-99e19e9ebebc\") " Oct 30 22:28:04.198185 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:04.198124 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85e8605a-0cd7-4eaf-84e2-99e19e9ebebc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85e8605a-0cd7-4eaf-84e2-99e19e9ebebc-workload-socket\") pod \"85e8605a-0cd7-4eaf-84e2-99e19e9ebebc\" (UID: \"85e8605a-0cd7-4eaf-84e2-99e19e9ebebc\") " Oct 30 22:28:04.198335 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:04.198221 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/85e8605a-0cd7-4eaf-84e2-99e19e9ebebc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85e8605a-0cd7-4eaf-84e2-99e19e9ebebc-istio-podinfo\") pod \"85e8605a-0cd7-4eaf-84e2-99e19e9ebebc\" (UID: \"85e8605a-0cd7-4eaf-84e2-99e19e9ebebc\") " Oct 30 22:28:04.198335 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:04.198284 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85e8605a-0cd7-4eaf-84e2-99e19e9ebebc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85e8605a-0cd7-4eaf-84e2-99e19e9ebebc-workload-certs\") pod \"85e8605a-0cd7-4eaf-84e2-99e19e9ebebc\" (UID: \"85e8605a-0cd7-4eaf-84e2-99e19e9ebebc\") " Oct 30 22:28:04.199142 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:04.198792 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85e8605a-0cd7-4eaf-84e2-99e19e9ebebc-workload-socket" pod "85e8605a-0cd7-4eaf-84e2-99e19e9ebebc" (UID: "85e8605a-0cd7-4eaf-84e2-99e19e9ebebc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:04.199142 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:04.198806 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85e8605a-0cd7-4eaf-84e2-99e19e9ebebc-credential-socket" pod "85e8605a-0cd7-4eaf-84e2-99e19e9ebebc" (UID: "85e8605a-0cd7-4eaf-84e2-99e19e9ebebc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:04.199142 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:04.199005 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85e8605a-0cd7-4eaf-84e2-99e19e9ebebc-workload-certs" pod "85e8605a-0cd7-4eaf-84e2-99e19e9ebebc" (UID: "85e8605a-0cd7-4eaf-84e2-99e19e9ebebc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:04.199142 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:04.199102 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85e8605a-0cd7-4eaf-84e2-99e19e9ebebc-istiod-ca-cert" pod "85e8605a-0cd7-4eaf-84e2-99e19e9ebebc" (UID: "85e8605a-0cd7-4eaf-84e2-99e19e9ebebc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:28:04.199716 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:04.199273 6720 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85e8605a-0cd7-4eaf-84e2-99e19e9ebebc-istiod-ca-cert\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:04.199716 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:04.199296 6720 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85e8605a-0cd7-4eaf-84e2-99e19e9ebebc-credential-socket\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:04.199716 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:04.199312 6720 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85e8605a-0cd7-4eaf-84e2-99e19e9ebebc-workload-socket\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:04.199716 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:04.199324 6720 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85e8605a-0cd7-4eaf-84e2-99e19e9ebebc-workload-certs\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:04.199716 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:04.199638 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85e8605a-0cd7-4eaf-84e2-99e19e9ebebc-istio-data" pod "85e8605a-0cd7-4eaf-84e2-99e19e9ebebc" (UID: "85e8605a-0cd7-4eaf-84e2-99e19e9ebebc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:04.200646 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:04.200618 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85e8605a-0cd7-4eaf-84e2-99e19e9ebebc-istio-token" pod "85e8605a-0cd7-4eaf-84e2-99e19e9ebebc" (UID: "85e8605a-0cd7-4eaf-84e2-99e19e9ebebc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:28:04.201078 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:04.201053 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85e8605a-0cd7-4eaf-84e2-99e19e9ebebc-istio-envoy" pod "85e8605a-0cd7-4eaf-84e2-99e19e9ebebc" (UID: "85e8605a-0cd7-4eaf-84e2-99e19e9ebebc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:04.201477 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:04.201448 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85e8605a-0cd7-4eaf-84e2-99e19e9ebebc-kube-api-access-74wvk" pod "85e8605a-0cd7-4eaf-84e2-99e19e9ebebc" (UID: "85e8605a-0cd7-4eaf-84e2-99e19e9ebebc"). InnerVolumeSpecName "kube-api-access-74wvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:28:04.202012 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:04.201987 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/85e8605a-0cd7-4eaf-84e2-99e19e9ebebc-istio-podinfo" pod "85e8605a-0cd7-4eaf-84e2-99e19e9ebebc" (UID: "85e8605a-0cd7-4eaf-84e2-99e19e9ebebc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:28:04.300965 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:04.300929 6720 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85e8605a-0cd7-4eaf-84e2-99e19e9ebebc-istio-podinfo\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:04.300965 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:04.300964 6720 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85e8605a-0cd7-4eaf-84e2-99e19e9ebebc-istio-token\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:04.300965 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:04.300974 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74wvk\" (UniqueName: \"kubernetes.io/projected/85e8605a-0cd7-4eaf-84e2-99e19e9ebebc-kube-api-access-74wvk\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:04.301450 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:04.300984 6720 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85e8605a-0cd7-4eaf-84e2-99e19e9ebebc-istio-data\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:04.301450 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:04.300993 6720 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85e8605a-0cd7-4eaf-84e2-99e19e9ebebc-istio-envoy\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:04.515675 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:04.515589 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:04.661748 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:04.661709 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:04.953458 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:04.953418 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:04.988995 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:04.988969 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-7v8vz" Oct 30 22:28:04.989183 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:04.988995 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-7v8vz" event={"ID":"85e8605a-0cd7-4eaf-84e2-99e19e9ebebc","Type":"ContainerDied","Data":"6248cd3550cd5b5f33e3483ef1fae06eaf8d4a7d0074b36604e13d1fc4afed70"} Oct 30 22:28:04.989183 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:04.989063 6720 scope.go:122] "RemoveContainer" containerID="81ddd0d3b4e6e80f74ae75ff158a5885e9abe291aee626002650b947fe949ccf" Oct 30 22:28:05.017200 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:05.017135 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-7v8vz"] Oct 30 22:28:05.021162 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:05.021130 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-2b2e72-openshift-default-bc46c6865-7v8vz"] Oct 30 22:28:05.125411 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:05.115654 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:05.509477 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:05.508845 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:06.003478 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:06.003435 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85e8605a-0cd7-4eaf-84e2-99e19e9ebebc" path="/var/lib/kubelet/pods/85e8605a-0cd7-4eaf-84e2-99e19e9ebebc/volumes" Oct 30 22:28:06.308464 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:06.299784 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:07.104638 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:07.104600 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:08.225011 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:08.224967 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:08.412453 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:08.408284 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:09.182427 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:09.182369 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:09.478545 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:09.478458 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:09.663409 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:09.635885 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:10.112245 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:10.112211 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:10.406307 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:28:10.406222 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 22:28:10.851007 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:10.850875 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:11.554582 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:11.554537 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:12.205542 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:12.205482 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:12.918450 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:12.916167 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:13.646965 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:13.646930 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:14.325931 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:14.324280 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:15.015227 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:15.015163 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:15.654562 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:15.653470 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:16.389430 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:16.389372 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:17.133189 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:17.133148 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:17.717256 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:17.709462 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:18.509886 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:18.509828 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:19.246056 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:19.246017 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:19.907481 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:19.902185 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:20.586612 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:20.586577 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:21.220976 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:21.220927 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:21.948376 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:21.948338 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:22.768482 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:22.768428 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:23.335768 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:23.335713 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:23.973291 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:23.973255 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:24.674845 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:24.674791 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:25.222762 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:25.222726 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f60026-564db895d5-nljrz"] Oct 30 22:28:25.223875 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:25.223852 6720 cpu_manager.go:414] "containerMap: removing container" podUID="85e8605a-0cd7-4eaf-84e2-99e19e9ebebc" containerName="istio-proxy" Oct 30 22:28:25.223943 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:25.223880 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85e8605a-0cd7-4eaf-84e2-99e19e9ebebc" containerName="istio-proxy" Oct 30 22:28:25.224079 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:25.224066 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="85e8605a-0cd7-4eaf-84e2-99e19e9ebebc" containerName="istio-proxy" Oct 30 22:28:25.229954 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:25.229927 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f60026-564db895d5-nljrz" Oct 30 22:28:25.244837 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:25.244796 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-f60026-serving-cert\"" Oct 30 22:28:25.257445 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:25.257404 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f60026-564db895d5-nljrz"] Oct 30 22:28:25.297013 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:25.296977 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e1f3f99-8520-4097-b948-3ca747b75ec9-maas-api-tls\") pod \"maas-api-e2e-inf-a-f60026-564db895d5-nljrz\" (UID: \"0e1f3f99-8520-4097-b948-3ca747b75ec9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f60026-564db895d5-nljrz" Oct 30 22:28:25.297221 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:25.297039 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9twd\" (UniqueName: \"kubernetes.io/projected/0e1f3f99-8520-4097-b948-3ca747b75ec9-kube-api-access-x9twd\") pod \"maas-api-e2e-inf-a-f60026-564db895d5-nljrz\" (UID: \"0e1f3f99-8520-4097-b948-3ca747b75ec9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f60026-564db895d5-nljrz" Oct 30 22:28:25.297221 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:25.297176 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e1f3f99-8520-4097-b948-3ca747b75ec9-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f60026-564db895d5-nljrz\" (UID: \"0e1f3f99-8520-4097-b948-3ca747b75ec9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f60026-564db895d5-nljrz" Oct 30 22:28:25.297339 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:25.297325 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e1f3f99-8520-4097-b948-3ca747b75ec9-metrics-certs\") pod \"maas-api-e2e-inf-a-f60026-564db895d5-nljrz\" (UID: \"0e1f3f99-8520-4097-b948-3ca747b75ec9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f60026-564db895d5-nljrz" Oct 30 22:28:25.319217 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:25.319164 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:25.412412 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:25.399403 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e1f3f99-8520-4097-b948-3ca747b75ec9-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f60026-564db895d5-nljrz\" (UID: \"0e1f3f99-8520-4097-b948-3ca747b75ec9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f60026-564db895d5-nljrz" Oct 30 22:28:25.412412 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:25.399654 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e1f3f99-8520-4097-b948-3ca747b75ec9-metrics-certs\") pod \"maas-api-e2e-inf-a-f60026-564db895d5-nljrz\" (UID: \"0e1f3f99-8520-4097-b948-3ca747b75ec9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f60026-564db895d5-nljrz" Oct 30 22:28:25.419660 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:25.414099 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e1f3f99-8520-4097-b948-3ca747b75ec9-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f60026-564db895d5-nljrz\" (UID: \"0e1f3f99-8520-4097-b948-3ca747b75ec9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f60026-564db895d5-nljrz" Oct 30 22:28:25.422587 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:25.422523 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e1f3f99-8520-4097-b948-3ca747b75ec9-metrics-certs\") pod \"maas-api-e2e-inf-a-f60026-564db895d5-nljrz\" (UID: \"0e1f3f99-8520-4097-b948-3ca747b75ec9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f60026-564db895d5-nljrz" Oct 30 22:28:25.431565 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:25.423602 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e1f3f99-8520-4097-b948-3ca747b75ec9-maas-api-tls\") pod \"maas-api-e2e-inf-a-f60026-564db895d5-nljrz\" (UID: \"0e1f3f99-8520-4097-b948-3ca747b75ec9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f60026-564db895d5-nljrz" Oct 30 22:28:25.431565 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:25.423714 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9twd\" (UniqueName: \"kubernetes.io/projected/0e1f3f99-8520-4097-b948-3ca747b75ec9-kube-api-access-x9twd\") pod \"maas-api-e2e-inf-a-f60026-564db895d5-nljrz\" (UID: \"0e1f3f99-8520-4097-b948-3ca747b75ec9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f60026-564db895d5-nljrz" Oct 30 22:28:25.442069 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:25.441982 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e1f3f99-8520-4097-b948-3ca747b75ec9-maas-api-tls\") pod \"maas-api-e2e-inf-a-f60026-564db895d5-nljrz\" (UID: \"0e1f3f99-8520-4097-b948-3ca747b75ec9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f60026-564db895d5-nljrz" Oct 30 22:28:25.449059 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:25.449030 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9twd\" (UniqueName: \"kubernetes.io/projected/0e1f3f99-8520-4097-b948-3ca747b75ec9-kube-api-access-x9twd\") pod \"maas-api-e2e-inf-a-f60026-564db895d5-nljrz\" (UID: \"0e1f3f99-8520-4097-b948-3ca747b75ec9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f60026-564db895d5-nljrz" Oct 30 22:28:25.462276 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:25.462243 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:25.556943 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:25.556844 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f60026-564db895d5-nljrz" Oct 30 22:28:25.920036 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:25.920004 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f60026-564db895d5-nljrz"] Oct 30 22:28:25.924597 ip-10-0-137-181 kubenswrapper[6720]: W1030 22:28:25.924567 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e1f3f99_8520_4097_b948_3ca747b75ec9.slice/crio-69f1d640e19b55defd01ae7e5abdeaf68a3d8f56d04eb4ccfa17fe985a771bd1 WatchSource:0}: Error finding container 69f1d640e19b55defd01ae7e5abdeaf68a3d8f56d04eb4ccfa17fe985a771bd1: Status 404 returned error can't find the container with id 69f1d640e19b55defd01ae7e5abdeaf68a3d8f56d04eb4ccfa17fe985a771bd1 Oct 30 22:28:26.179254 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:26.177990 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:26.269435 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:26.269403 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f60026-564db895d5-nljrz" event={"ID":"0e1f3f99-8520-4097-b948-3ca747b75ec9","Type":"ContainerStarted","Data":"69f1d640e19b55defd01ae7e5abdeaf68a3d8f56d04eb4ccfa17fe985a771bd1"} Oct 30 22:28:26.841520 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:26.838875 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:27.279256 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:27.279213 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f60026-564db895d5-nljrz" event={"ID":"0e1f3f99-8520-4097-b948-3ca747b75ec9","Type":"ContainerStarted","Data":"0d0c7f42eb53f8f3598c1ddb8216787a9dabcf85abd0dd340b02d9aedad9c7ef"} Oct 30 22:28:27.279681 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:27.279297 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f60026-564db895d5-nljrz" Oct 30 22:28:27.302102 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:27.302047 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f60026-564db895d5-nljrz" podStartSLOduration=1.8563326999999998 podStartE2EDuration="2.302033107s" totalImagesPullingTime="445.700407ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:28:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:28:27.300685482 +0000 UTC m=+2873.471650665" watchObservedRunningTime="2026-10-30 22:28:27.302033107 +0000 UTC m=+2873.472998288" Oct 30 22:28:27.528007 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:27.527962 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:28.167899 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:28.167101 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:28.945526 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:28.935237 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:29.722408 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:29.718934 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:30.012522 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:30.012442 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:30.344549 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:30.342353 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:30.499329 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:30.499293 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:30.981408 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:30.969620 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:31.652587 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:31.652523 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:32.237756 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:32.237721 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:32.874425 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:32.871696 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:33.292033 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:33.291954 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f60026-564db895d5-nljrz" Oct 30 22:28:33.560812 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:33.560717 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:34.244411 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:34.243615 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:34.923412 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:34.919547 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:35.567178 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:35.567138 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:36.334425 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:36.332663 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:37.024982 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:37.024929 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:37.690943 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:37.690903 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:38.376407 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:38.376349 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:39.006932 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:39.006888 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:39.631108 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:39.631073 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:40.254267 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:40.254225 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:40.455495 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:28:40.455418 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 22:28:41.100741 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:41.100704 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:42.104511 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:42.104474 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:43.102094 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:43.102056 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:43.822728 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:43.822513 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:44.713009 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:44.712912 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:45.568429 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:45.568343 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:46.216311 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:46.216252 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:47.033305 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:47.033258 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:47.693114 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:47.693038 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:48.338284 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:48.338232 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:48.883224 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:48.883183 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:49.043404 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:49.043339 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:49.238951 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:49.238871 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:49.678289 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:49.678248 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:49.958177 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:49.958081 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:50.134801 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:50.134768 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:50.535069 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:50.535024 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:51.218588 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:51.218552 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:51.921288 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:51.921246 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:52.704844 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:52.704802 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:53.378026 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:53.377993 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:53.711405 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:53.711295 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:53.852149 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:53.852117 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:54.105371 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:54.105335 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:54.928605 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:54.928570 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:55.210421 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:55.210315 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:55.733950 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:55.733905 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:56.498415 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:56.491878 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:57.333780 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:57.333746 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:57.960489 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:57.960448 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:58.120188 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:58.120157 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-vnctc"] Oct 30 22:28:58.126036 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:58.126014 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-vnctc" Oct 30 22:28:58.142179 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:58.142153 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-vnctc"] Oct 30 22:28:58.282329 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:58.282229 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/961c219a-a80f-46eb-ab6c-2920a28419e2-credential-socket\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-vnctc\" (UID: \"961c219a-a80f-46eb-ab6c-2920a28419e2\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-vnctc" Oct 30 22:28:58.282543 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:58.282430 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/961c219a-a80f-46eb-ab6c-2920a28419e2-workload-certs\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-vnctc\" (UID: \"961c219a-a80f-46eb-ab6c-2920a28419e2\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-vnctc" Oct 30 22:28:58.282543 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:58.282463 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/961c219a-a80f-46eb-ab6c-2920a28419e2-istio-token\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-vnctc\" (UID: \"961c219a-a80f-46eb-ab6c-2920a28419e2\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-vnctc" Oct 30 22:28:58.282543 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:58.282489 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/961c219a-a80f-46eb-ab6c-2920a28419e2-istio-envoy\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-vnctc\" (UID: \"961c219a-a80f-46eb-ab6c-2920a28419e2\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-vnctc" Oct 30 22:28:58.282689 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:58.282617 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/961c219a-a80f-46eb-ab6c-2920a28419e2-workload-socket\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-vnctc\" (UID: \"961c219a-a80f-46eb-ab6c-2920a28419e2\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-vnctc" Oct 30 22:28:58.282689 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:58.282658 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f5k4\" (UniqueName: \"kubernetes.io/projected/961c219a-a80f-46eb-ab6c-2920a28419e2-kube-api-access-8f5k4\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-vnctc\" (UID: \"961c219a-a80f-46eb-ab6c-2920a28419e2\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-vnctc" Oct 30 22:28:58.282689 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:58.282682 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/961c219a-a80f-46eb-ab6c-2920a28419e2-istio-podinfo\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-vnctc\" (UID: \"961c219a-a80f-46eb-ab6c-2920a28419e2\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-vnctc" Oct 30 22:28:58.282813 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:58.282804 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/961c219a-a80f-46eb-ab6c-2920a28419e2-istio-data\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-vnctc\" (UID: \"961c219a-a80f-46eb-ab6c-2920a28419e2\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-vnctc" Oct 30 22:28:58.282874 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:58.282861 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/961c219a-a80f-46eb-ab6c-2920a28419e2-istiod-ca-cert\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-vnctc\" (UID: \"961c219a-a80f-46eb-ab6c-2920a28419e2\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-vnctc" Oct 30 22:28:58.384144 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:58.384106 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/961c219a-a80f-46eb-ab6c-2920a28419e2-istio-data\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-vnctc\" (UID: \"961c219a-a80f-46eb-ab6c-2920a28419e2\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-vnctc" Oct 30 22:28:58.384329 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:58.384213 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/961c219a-a80f-46eb-ab6c-2920a28419e2-istiod-ca-cert\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-vnctc\" (UID: \"961c219a-a80f-46eb-ab6c-2920a28419e2\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-vnctc" Oct 30 22:28:58.384329 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:58.384278 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/961c219a-a80f-46eb-ab6c-2920a28419e2-credential-socket\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-vnctc\" (UID: \"961c219a-a80f-46eb-ab6c-2920a28419e2\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-vnctc" Oct 30 22:28:58.384496 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:58.384455 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/961c219a-a80f-46eb-ab6c-2920a28419e2-workload-certs\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-vnctc\" (UID: \"961c219a-a80f-46eb-ab6c-2920a28419e2\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-vnctc" Oct 30 22:28:58.384549 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:58.384521 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/961c219a-a80f-46eb-ab6c-2920a28419e2-istio-token\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-vnctc\" (UID: \"961c219a-a80f-46eb-ab6c-2920a28419e2\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-vnctc" Oct 30 22:28:58.384599 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:58.384554 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/961c219a-a80f-46eb-ab6c-2920a28419e2-istio-envoy\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-vnctc\" (UID: \"961c219a-a80f-46eb-ab6c-2920a28419e2\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-vnctc" Oct 30 22:28:58.384644 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:58.384608 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/961c219a-a80f-46eb-ab6c-2920a28419e2-workload-socket\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-vnctc\" (UID: \"961c219a-a80f-46eb-ab6c-2920a28419e2\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-vnctc" Oct 30 22:28:58.384644 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:58.384633 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f5k4\" (UniqueName: \"kubernetes.io/projected/961c219a-a80f-46eb-ab6c-2920a28419e2-kube-api-access-8f5k4\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-vnctc\" (UID: \"961c219a-a80f-46eb-ab6c-2920a28419e2\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-vnctc" Oct 30 22:28:58.384727 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:58.384664 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/961c219a-a80f-46eb-ab6c-2920a28419e2-istio-podinfo\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-vnctc\" (UID: \"961c219a-a80f-46eb-ab6c-2920a28419e2\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-vnctc" Oct 30 22:28:58.384727 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:58.384699 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/961c219a-a80f-46eb-ab6c-2920a28419e2-credential-socket\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-vnctc\" (UID: \"961c219a-a80f-46eb-ab6c-2920a28419e2\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-vnctc" Oct 30 22:28:58.384988 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:58.384955 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/961c219a-a80f-46eb-ab6c-2920a28419e2-istiod-ca-cert\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-vnctc\" (UID: \"961c219a-a80f-46eb-ab6c-2920a28419e2\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-vnctc" Oct 30 22:28:58.385056 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:58.384605 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/961c219a-a80f-46eb-ab6c-2920a28419e2-istio-data\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-vnctc\" (UID: \"961c219a-a80f-46eb-ab6c-2920a28419e2\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-vnctc" Oct 30 22:28:58.385118 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:58.385041 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/961c219a-a80f-46eb-ab6c-2920a28419e2-workload-socket\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-vnctc\" (UID: \"961c219a-a80f-46eb-ab6c-2920a28419e2\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-vnctc" Oct 30 22:28:58.385173 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:58.385128 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/961c219a-a80f-46eb-ab6c-2920a28419e2-workload-certs\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-vnctc\" (UID: \"961c219a-a80f-46eb-ab6c-2920a28419e2\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-vnctc" Oct 30 22:28:58.387161 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:58.387136 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/961c219a-a80f-46eb-ab6c-2920a28419e2-istio-envoy\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-vnctc\" (UID: \"961c219a-a80f-46eb-ab6c-2920a28419e2\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-vnctc" Oct 30 22:28:58.387492 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:58.387475 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/961c219a-a80f-46eb-ab6c-2920a28419e2-istio-podinfo\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-vnctc\" (UID: \"961c219a-a80f-46eb-ab6c-2920a28419e2\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-vnctc" Oct 30 22:28:58.397051 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:58.397022 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/961c219a-a80f-46eb-ab6c-2920a28419e2-istio-token\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-vnctc\" (UID: \"961c219a-a80f-46eb-ab6c-2920a28419e2\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-vnctc" Oct 30 22:28:58.397673 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:58.397649 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f5k4\" (UniqueName: \"kubernetes.io/projected/961c219a-a80f-46eb-ab6c-2920a28419e2-kube-api-access-8f5k4\") pod \"e2e-inf-a-120cb1-openshift-default-d97c644bf-vnctc\" (UID: \"961c219a-a80f-46eb-ab6c-2920a28419e2\") " pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-vnctc" Oct 30 22:28:58.443633 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:58.443595 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-vnctc" Oct 30 22:28:58.713759 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:58.712155 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:58.764134 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:58.764109 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-vnctc"] Oct 30 22:28:58.765962 ip-10-0-137-181 kubenswrapper[6720]: W1030 22:28:58.765924 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod961c219a_a80f_46eb_ab6c_2920a28419e2.slice/crio-a99c17bfa741c253e12b2a7741e282bce2da05b13978aeec1e82cb7cc69437f1 WatchSource:0}: Error finding container a99c17bfa741c253e12b2a7741e282bce2da05b13978aeec1e82cb7cc69437f1: Status 404 returned error can't find the container with id a99c17bfa741c253e12b2a7741e282bce2da05b13978aeec1e82cb7cc69437f1 Oct 30 22:28:58.768972 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:58.768940 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:28:58.769078 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:58.769021 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:28:58.769078 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:58.769069 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:28:59.342371 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:59.342317 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:28:59.636702 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:59.636667 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-vnctc" event={"ID":"961c219a-a80f-46eb-ab6c-2920a28419e2","Type":"ContainerStarted","Data":"093ba049dfdd3a8f1194b04b2e1b09bc49974a3d64ca79369d6482d8f22859ce"} Oct 30 22:28:59.636904 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:59.636721 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-vnctc" event={"ID":"961c219a-a80f-46eb-ab6c-2920a28419e2","Type":"ContainerStarted","Data":"a99c17bfa741c253e12b2a7741e282bce2da05b13978aeec1e82cb7cc69437f1"} Oct 30 22:28:59.669009 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:28:59.668925 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-vnctc" podStartSLOduration=1.668904721 podStartE2EDuration="1.668904721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:28:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:28:59.663408201 +0000 UTC m=+2905.834373385" watchObservedRunningTime="2026-10-30 22:28:59.668904721 +0000 UTC m=+2905.839869904" Oct 30 22:29:00.143376 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:00.142889 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:00.469610 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:00.463344 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-vnctc" Oct 30 22:29:00.470583 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:00.470542 6720 patch_prober.go:29] interesting pod/e2e-inf-a-120cb1-openshift-default-d97c644bf-vnctc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 30 22:29:00.470747 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:00.470611 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-vnctc" podUID="961c219a-a80f-46eb-ab6c-2920a28419e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 30 22:29:00.932519 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:00.932487 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:01.444270 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:01.444228 6720 patch_prober.go:29] interesting pod/e2e-inf-a-120cb1-openshift-default-d97c644bf-vnctc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 30 22:29:01.444507 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:01.444305 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-vnctc" podUID="961c219a-a80f-46eb-ab6c-2920a28419e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 30 22:29:01.723091 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:01.723054 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:02.464412 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:02.454019 6720 patch_prober.go:29] interesting pod/e2e-inf-a-120cb1-openshift-default-d97c644bf-vnctc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 30 22:29:02.464412 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:02.454095 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-vnctc" podUID="961c219a-a80f-46eb-ab6c-2920a28419e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 30 22:29:02.596410 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:02.591595 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:03.395410 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:03.391237 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:03.445420 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:03.444762 6720 patch_prober.go:29] interesting pod/e2e-inf-a-120cb1-openshift-default-d97c644bf-vnctc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 30 22:29:03.445420 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:03.444825 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-vnctc" podUID="961c219a-a80f-46eb-ab6c-2920a28419e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 30 22:29:03.680996 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:03.680915 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-vvpgw"] Oct 30 22:29:03.685918 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:03.685891 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-vvpgw" Oct 30 22:29:03.690596 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:03.690569 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-28d39b-openshift-default-dockercfg-qsmx6\"" Oct 30 22:29:03.701462 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:03.701439 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-vvpgw"] Oct 30 22:29:03.802349 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:03.802314 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ea7cd43-6136-4a44-921d-65fbf8c39f76-istio-token\") pod \"e2e-inf-a-28d39b-openshift-default-5544d7c55f-vvpgw\" (UID: \"1ea7cd43-6136-4a44-921d-65fbf8c39f76\") " pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-vvpgw" Oct 30 22:29:03.802583 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:03.802432 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ea7cd43-6136-4a44-921d-65fbf8c39f76-istio-podinfo\") pod \"e2e-inf-a-28d39b-openshift-default-5544d7c55f-vvpgw\" (UID: \"1ea7cd43-6136-4a44-921d-65fbf8c39f76\") " pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-vvpgw" Oct 30 22:29:03.802583 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:03.802457 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ea7cd43-6136-4a44-921d-65fbf8c39f76-istiod-ca-cert\") pod \"e2e-inf-a-28d39b-openshift-default-5544d7c55f-vvpgw\" (UID: \"1ea7cd43-6136-4a44-921d-65fbf8c39f76\") " pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-vvpgw" Oct 30 22:29:03.802583 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:03.802478 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ea7cd43-6136-4a44-921d-65fbf8c39f76-workload-certs\") pod \"e2e-inf-a-28d39b-openshift-default-5544d7c55f-vvpgw\" (UID: \"1ea7cd43-6136-4a44-921d-65fbf8c39f76\") " pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-vvpgw" Oct 30 22:29:03.802731 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:03.802649 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ea7cd43-6136-4a44-921d-65fbf8c39f76-istio-data\") pod \"e2e-inf-a-28d39b-openshift-default-5544d7c55f-vvpgw\" (UID: \"1ea7cd43-6136-4a44-921d-65fbf8c39f76\") " pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-vvpgw" Oct 30 22:29:03.802731 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:03.802690 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ea7cd43-6136-4a44-921d-65fbf8c39f76-credential-socket\") pod \"e2e-inf-a-28d39b-openshift-default-5544d7c55f-vvpgw\" (UID: \"1ea7cd43-6136-4a44-921d-65fbf8c39f76\") " pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-vvpgw" Oct 30 22:29:03.802815 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:03.802761 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ea7cd43-6136-4a44-921d-65fbf8c39f76-workload-socket\") pod \"e2e-inf-a-28d39b-openshift-default-5544d7c55f-vvpgw\" (UID: \"1ea7cd43-6136-4a44-921d-65fbf8c39f76\") " pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-vvpgw" Oct 30 22:29:03.802850 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:03.802831 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq2n2\" (UniqueName: \"kubernetes.io/projected/1ea7cd43-6136-4a44-921d-65fbf8c39f76-kube-api-access-sq2n2\") pod \"e2e-inf-a-28d39b-openshift-default-5544d7c55f-vvpgw\" (UID: \"1ea7cd43-6136-4a44-921d-65fbf8c39f76\") " pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-vvpgw" Oct 30 22:29:03.802888 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:03.802872 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ea7cd43-6136-4a44-921d-65fbf8c39f76-istio-envoy\") pod \"e2e-inf-a-28d39b-openshift-default-5544d7c55f-vvpgw\" (UID: \"1ea7cd43-6136-4a44-921d-65fbf8c39f76\") " pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-vvpgw" Oct 30 22:29:03.904833 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:03.904798 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ea7cd43-6136-4a44-921d-65fbf8c39f76-istio-envoy\") pod \"e2e-inf-a-28d39b-openshift-default-5544d7c55f-vvpgw\" (UID: \"1ea7cd43-6136-4a44-921d-65fbf8c39f76\") " pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-vvpgw" Oct 30 22:29:03.905041 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:03.904982 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ea7cd43-6136-4a44-921d-65fbf8c39f76-istio-token\") pod \"e2e-inf-a-28d39b-openshift-default-5544d7c55f-vvpgw\" (UID: \"1ea7cd43-6136-4a44-921d-65fbf8c39f76\") " pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-vvpgw" Oct 30 22:29:03.905088 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:03.905057 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ea7cd43-6136-4a44-921d-65fbf8c39f76-istio-podinfo\") pod \"e2e-inf-a-28d39b-openshift-default-5544d7c55f-vvpgw\" (UID: \"1ea7cd43-6136-4a44-921d-65fbf8c39f76\") " pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-vvpgw" Oct 30 22:29:03.905226 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:03.905200 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ea7cd43-6136-4a44-921d-65fbf8c39f76-istiod-ca-cert\") pod \"e2e-inf-a-28d39b-openshift-default-5544d7c55f-vvpgw\" (UID: \"1ea7cd43-6136-4a44-921d-65fbf8c39f76\") " pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-vvpgw" Oct 30 22:29:03.905290 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:03.905247 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ea7cd43-6136-4a44-921d-65fbf8c39f76-workload-certs\") pod \"e2e-inf-a-28d39b-openshift-default-5544d7c55f-vvpgw\" (UID: \"1ea7cd43-6136-4a44-921d-65fbf8c39f76\") " pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-vvpgw" Oct 30 22:29:03.905348 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:03.905322 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ea7cd43-6136-4a44-921d-65fbf8c39f76-istio-data\") pod \"e2e-inf-a-28d39b-openshift-default-5544d7c55f-vvpgw\" (UID: \"1ea7cd43-6136-4a44-921d-65fbf8c39f76\") " pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-vvpgw" Oct 30 22:29:03.905424 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:03.905366 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ea7cd43-6136-4a44-921d-65fbf8c39f76-credential-socket\") pod \"e2e-inf-a-28d39b-openshift-default-5544d7c55f-vvpgw\" (UID: \"1ea7cd43-6136-4a44-921d-65fbf8c39f76\") " pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-vvpgw" Oct 30 22:29:03.905485 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:03.905465 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ea7cd43-6136-4a44-921d-65fbf8c39f76-workload-socket\") pod \"e2e-inf-a-28d39b-openshift-default-5544d7c55f-vvpgw\" (UID: \"1ea7cd43-6136-4a44-921d-65fbf8c39f76\") " pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-vvpgw" Oct 30 22:29:03.905554 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:03.905537 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq2n2\" (UniqueName: \"kubernetes.io/projected/1ea7cd43-6136-4a44-921d-65fbf8c39f76-kube-api-access-sq2n2\") pod \"e2e-inf-a-28d39b-openshift-default-5544d7c55f-vvpgw\" (UID: \"1ea7cd43-6136-4a44-921d-65fbf8c39f76\") " pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-vvpgw" Oct 30 22:29:03.905758 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:03.905733 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ea7cd43-6136-4a44-921d-65fbf8c39f76-workload-certs\") pod \"e2e-inf-a-28d39b-openshift-default-5544d7c55f-vvpgw\" (UID: \"1ea7cd43-6136-4a44-921d-65fbf8c39f76\") " pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-vvpgw" Oct 30 22:29:03.906021 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:03.905999 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ea7cd43-6136-4a44-921d-65fbf8c39f76-istio-data\") pod \"e2e-inf-a-28d39b-openshift-default-5544d7c55f-vvpgw\" (UID: \"1ea7cd43-6136-4a44-921d-65fbf8c39f76\") " pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-vvpgw" Oct 30 22:29:03.906155 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:03.906016 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ea7cd43-6136-4a44-921d-65fbf8c39f76-istiod-ca-cert\") pod \"e2e-inf-a-28d39b-openshift-default-5544d7c55f-vvpgw\" (UID: \"1ea7cd43-6136-4a44-921d-65fbf8c39f76\") " pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-vvpgw" Oct 30 22:29:03.906155 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:03.906031 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ea7cd43-6136-4a44-921d-65fbf8c39f76-credential-socket\") pod \"e2e-inf-a-28d39b-openshift-default-5544d7c55f-vvpgw\" (UID: \"1ea7cd43-6136-4a44-921d-65fbf8c39f76\") " pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-vvpgw" Oct 30 22:29:03.906300 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:03.906043 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ea7cd43-6136-4a44-921d-65fbf8c39f76-workload-socket\") pod \"e2e-inf-a-28d39b-openshift-default-5544d7c55f-vvpgw\" (UID: \"1ea7cd43-6136-4a44-921d-65fbf8c39f76\") " pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-vvpgw" Oct 30 22:29:03.907429 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:03.907405 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ea7cd43-6136-4a44-921d-65fbf8c39f76-istio-envoy\") pod \"e2e-inf-a-28d39b-openshift-default-5544d7c55f-vvpgw\" (UID: \"1ea7cd43-6136-4a44-921d-65fbf8c39f76\") " pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-vvpgw" Oct 30 22:29:03.907746 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:03.907724 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ea7cd43-6136-4a44-921d-65fbf8c39f76-istio-podinfo\") pod \"e2e-inf-a-28d39b-openshift-default-5544d7c55f-vvpgw\" (UID: \"1ea7cd43-6136-4a44-921d-65fbf8c39f76\") " pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-vvpgw" Oct 30 22:29:03.917636 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:03.917610 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ea7cd43-6136-4a44-921d-65fbf8c39f76-istio-token\") pod \"e2e-inf-a-28d39b-openshift-default-5544d7c55f-vvpgw\" (UID: \"1ea7cd43-6136-4a44-921d-65fbf8c39f76\") " pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-vvpgw" Oct 30 22:29:03.917859 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:03.917841 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq2n2\" (UniqueName: \"kubernetes.io/projected/1ea7cd43-6136-4a44-921d-65fbf8c39f76-kube-api-access-sq2n2\") pod \"e2e-inf-a-28d39b-openshift-default-5544d7c55f-vvpgw\" (UID: \"1ea7cd43-6136-4a44-921d-65fbf8c39f76\") " pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-vvpgw" Oct 30 22:29:04.005171 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:04.005085 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-vvpgw" Oct 30 22:29:04.117445 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:04.116333 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:04.380881 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:04.380820 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-vvpgw"] Oct 30 22:29:04.381130 ip-10-0-137-181 kubenswrapper[6720]: W1030 22:29:04.381103 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ea7cd43_6136_4a44_921d_65fbf8c39f76.slice/crio-863ed938305bce8862cc7c55e0487a38cc328a1c511c0ae0ce1a35c9109b7c10 WatchSource:0}: Error finding container 863ed938305bce8862cc7c55e0487a38cc328a1c511c0ae0ce1a35c9109b7c10: Status 404 returned error can't find the container with id 863ed938305bce8862cc7c55e0487a38cc328a1c511c0ae0ce1a35c9109b7c10 Oct 30 22:29:04.384220 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:04.384189 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:29:04.384318 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:04.384253 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:29:04.384318 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:04.384282 6720 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 30 22:29:04.457411 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:04.451645 6720 patch_prober.go:29] interesting pod/e2e-inf-a-120cb1-openshift-default-d97c644bf-vnctc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 30 22:29:04.457411 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:04.451981 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-vnctc" podUID="961c219a-a80f-46eb-ab6c-2920a28419e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 30 22:29:04.701882 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:04.701789 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-vvpgw" event={"ID":"1ea7cd43-6136-4a44-921d-65fbf8c39f76","Type":"ContainerStarted","Data":"98f6ee11f0ee603c47f6dae9e5a65e88b1d9906f5412f67425e2e9980f6957c2"} Oct 30 22:29:04.701882 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:04.701845 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-vvpgw" event={"ID":"1ea7cd43-6136-4a44-921d-65fbf8c39f76","Type":"ContainerStarted","Data":"863ed938305bce8862cc7c55e0487a38cc328a1c511c0ae0ce1a35c9109b7c10"} Oct 30 22:29:04.728219 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:04.728156 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-vvpgw" podStartSLOduration=1.72812206 podStartE2EDuration="1.72812206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:29:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:29:04.725807185 +0000 UTC m=+2910.896772364" watchObservedRunningTime="2026-10-30 22:29:04.72812206 +0000 UTC m=+2910.899087243" Oct 30 22:29:04.964468 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:04.963201 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:05.005844 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:05.005802 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-vvpgw" Oct 30 22:29:05.010662 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:05.010296 6720 patch_prober.go:29] interesting pod/e2e-inf-a-28d39b-openshift-default-5544d7c55f-vvpgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 22:29:05.010662 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:05.010369 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-vvpgw" podUID="1ea7cd43-6136-4a44-921d-65fbf8c39f76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 22:29:05.580819 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:05.580778 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-vnctc" Oct 30 22:29:05.581938 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:05.581910 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-vnctc" Oct 30 22:29:05.583140 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:05.583118 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-vnctc" Oct 30 22:29:05.898412 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:05.879060 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:06.006498 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:06.006466 6720 patch_prober.go:29] interesting pod/e2e-inf-a-28d39b-openshift-default-5544d7c55f-vvpgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 22:29:06.006701 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:06.006528 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-vvpgw" podUID="1ea7cd43-6136-4a44-921d-65fbf8c39f76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 22:29:06.627569 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:06.627533 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:07.005940 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:07.005856 6720 patch_prober.go:29] interesting pod/e2e-inf-a-28d39b-openshift-default-5544d7c55f-vvpgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 22:29:07.005940 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:07.005913 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-vvpgw" podUID="1ea7cd43-6136-4a44-921d-65fbf8c39f76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 22:29:07.341578 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:07.341302 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:08.006371 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:08.006336 6720 patch_prober.go:29] interesting pod/e2e-inf-a-28d39b-openshift-default-5544d7c55f-vvpgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 22:29:08.006822 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:08.006422 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-vvpgw" podUID="1ea7cd43-6136-4a44-921d-65fbf8c39f76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 22:29:08.067410 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:08.067215 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:08.925441 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:08.922654 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:09.006447 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:09.006375 6720 patch_prober.go:29] interesting pod/e2e-inf-a-28d39b-openshift-default-5544d7c55f-vvpgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 22:29:09.006861 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:09.006461 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-vvpgw" podUID="1ea7cd43-6136-4a44-921d-65fbf8c39f76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 22:29:09.739053 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:09.739013 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:10.006190 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:10.006105 6720 patch_prober.go:29] interesting pod/e2e-inf-a-28d39b-openshift-default-5544d7c55f-vvpgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 22:29:10.006364 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:10.006182 6720 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-vvpgw" podUID="1ea7cd43-6136-4a44-921d-65fbf8c39f76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 22:29:10.406632 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:29:10.406587 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 22:29:10.543422 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:10.542110 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:11.179734 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:11.179701 6720 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-vvpgw" Oct 30 22:29:11.180425 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:11.180396 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-vvpgw" Oct 30 22:29:11.181605 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:11.181580 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-vvpgw" Oct 30 22:29:11.258412 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:11.254763 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:11.914107 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:11.912739 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:12.548412 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:12.537912 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:12.716982 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:12.716948 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:13.463743 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:13.463705 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:13.820463 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:13.820360 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:13.972837 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:13.972805 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:14.161770 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:14.161737 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:14.863107 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:14.863057 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:15.575101 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:15.569243 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:16.884357 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:16.884312 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:17.031011 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:17.030962 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:17.730353 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:17.730297 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:18.404250 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:18.404217 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:19.023279 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:19.023235 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:19.759442 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:19.758993 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:20.631972 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:20.631935 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:21.539410 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:21.523473 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:22.184124 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:22.184053 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:22.848410 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:22.847967 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:23.562411 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:23.559913 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:24.257111 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:24.256973 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:24.855699 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:24.855662 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:25.555939 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:25.555891 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:26.254758 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:26.254689 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:26.922406 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:26.921283 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:27.597520 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:27.592272 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:28.244284 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:28.244229 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:28.938528 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:28.936065 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:29.620489 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:29.620451 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:30.068157 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:30.068071 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:30.292429 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:30.292376 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:30.435378 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:30.435323 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:30.967411 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:30.963145 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:31.660126 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:31.660069 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:32.418473 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:32.413849 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:33.137541 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:33.137507 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:33.789024 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:33.788786 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:34.215640 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:34.215601 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:34.490002 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:34.489844 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:35.337128 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:35.332299 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:35.988061 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:35.988026 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:36.141872 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:36.141838 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:36.810795 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:36.810740 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:37.490959 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:37.490881 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:38.145044 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:38.145010 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:38.839378 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:38.839333 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:39.607821 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:39.602508 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:40.408015 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:29:40.407979 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 22:29:40.448215 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:40.448179 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:41.391419 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:41.390055 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:42.185122 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:42.185065 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:42.921108 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:42.921067 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:43.685138 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:43.685088 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:44.337368 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:44.337325 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:45.099225 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:45.099126 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:45.830488 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:45.830454 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:45.979898 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:45.979855 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:46.631510 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:46.631473 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:47.439059 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:47.430182 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:48.055833 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:48.055756 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:48.917167 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:48.917095 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:49.640061 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:49.639982 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:50.449477 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:50.449262 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:51.124848 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:51.124816 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:51.760697 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:51.760659 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:52.542525 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:52.542489 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:53.331430 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:53.329035 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:53.922914 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:53.921860 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:54.720496 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:54.720460 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:55.609058 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:55.609020 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:56.205751 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:56.205709 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:57.071109 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:57.071069 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:57.644407 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:57.642993 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:58.250501 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:58.250466 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:58.939061 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:58.939022 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:29:59.643246 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:29:59.635947 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:00.343517 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:00.343476 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:01.246772 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:01.246740 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:02.023864 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:02.023757 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:02.859469 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:02.859012 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:03.711947 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:03.711916 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:03.913699 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:03.913661 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:04.522928 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:04.522889 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:05.421156 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:05.421104 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:06.598124 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:06.598081 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:07.368776 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:07.367335 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:08.233293 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:08.233252 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:09.145777 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:09.142124 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:09.844701 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:09.844619 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:10.405620 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:30:10.405575 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 22:30:10.514580 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:10.513762 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:11.337437 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:11.334085 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:12.086887 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:12.086856 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:12.763739 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:12.763694 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:13.447683 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:13.447429 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:14.157166 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:14.156403 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:14.871688 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:14.871589 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:15.487436 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:15.484197 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:16.337531 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:16.335338 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:17.035133 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:17.035091 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:17.686097 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:17.686064 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:18.509544 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:18.509504 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:19.141999 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:19.141964 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:19.961774 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:19.959253 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:20.700596 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:20.700555 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:21.519109 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:21.519037 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:22.255664 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:22.254632 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:23.045081 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:23.045045 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:23.756066 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:23.756030 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:24.513427 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:24.513368 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:25.147248 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:25.147208 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:25.919269 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:25.919227 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:26.587332 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:26.587293 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:27.253479 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:27.253445 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:28.129408 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:28.122330 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:28.732673 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:28.732578 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:29.433248 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:29.433196 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:30.116763 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:30.116725 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:30.274116 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:30.274081 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:30.472928 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:30.472841 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:30.768160 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:30.768051 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:31.529019 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:31.528925 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:32.140819 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:32.140736 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:32.980304 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:32.980274 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:33.630680 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:33.630643 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:34.385864 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:34.385009 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:35.175157 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:35.175112 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:35.929202 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:35.929149 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:36.724748 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:36.724654 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:37.424046 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:37.424007 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:38.143992 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:38.143930 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:38.908414 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:38.893056 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:39.719561 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:39.719526 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:40.406768 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:30:40.406717 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 22:30:40.637548 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:40.637518 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:41.247303 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:41.246502 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:42.123981 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:42.123922 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:42.705409 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:42.705333 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:43.220476 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:43.219952 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:43.934964 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:43.934925 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:44.230263 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:44.230169 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-df200a/test-model-df200a-kserve-7fc978d98f-jc487"] Oct 30 22:30:44.346413 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:44.337418 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:44.859956 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:44.859918 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:45.610349 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:45.610313 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:46.237010 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:46.236910 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:47.064409 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:47.050435 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:47.755410 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:47.755068 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:48.529810 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:48.529770 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:49.237324 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:49.237274 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:50.023464 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:50.020091 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:50.663180 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:50.663132 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:51.320871 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:51.320841 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:52.018178 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:52.018108 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:52.649125 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:52.649091 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:53.433485 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:53.433449 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:53.745970 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:53.745888 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-58d5fa/test-model-58d5fa-kserve-746d5495f7-88wps"] Oct 30 22:30:53.822866 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:53.822817 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:54.459082 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:54.458986 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:55.514252 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:55.514217 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:56.276412 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:56.268472 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:57.153413 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:57.141685 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:58.055524 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:58.051015 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:58.762943 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:58.762878 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:30:59.270634 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:59.270587 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-df200a/test-model-df200a-kserve-7fc978d98f-jc487" podUID="3360dc0f-497d-4211-a73f-c5ac0ff8f204" containerName="main" containerID="cri-o://b9cc50e667f7a065cc2c30ba3bc6625d8c292eb015208a9980eba973d6b9d88e" gracePeriod=45 Oct 30 22:30:59.619310 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:59.619275 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-df200a/test-model-df200a-kserve-7fc978d98f-jc487" Oct 30 22:30:59.732094 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:59.732048 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3360dc0f-497d-4211-a73f-c5ac0ff8f204-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3360dc0f-497d-4211-a73f-c5ac0ff8f204-tmp-dir\") pod \"3360dc0f-497d-4211-a73f-c5ac0ff8f204\" (UID: \"3360dc0f-497d-4211-a73f-c5ac0ff8f204\") " Oct 30 22:30:59.732299 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:59.732227 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3360dc0f-497d-4211-a73f-c5ac0ff8f204-home\" (UniqueName: \"kubernetes.io/empty-dir/3360dc0f-497d-4211-a73f-c5ac0ff8f204-home\") pod \"3360dc0f-497d-4211-a73f-c5ac0ff8f204\" (UID: \"3360dc0f-497d-4211-a73f-c5ac0ff8f204\") " Oct 30 22:30:59.732403 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:59.732321 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3360dc0f-497d-4211-a73f-c5ac0ff8f204-kube-api-access-q98jl\" (UniqueName: \"kubernetes.io/projected/3360dc0f-497d-4211-a73f-c5ac0ff8f204-kube-api-access-q98jl\") pod \"3360dc0f-497d-4211-a73f-c5ac0ff8f204\" (UID: \"3360dc0f-497d-4211-a73f-c5ac0ff8f204\") " Oct 30 22:30:59.732403 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:59.732325 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3360dc0f-497d-4211-a73f-c5ac0ff8f204-tmp-dir" pod "3360dc0f-497d-4211-a73f-c5ac0ff8f204" (UID: "3360dc0f-497d-4211-a73f-c5ac0ff8f204"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:30:59.732403 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:59.732362 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3360dc0f-497d-4211-a73f-c5ac0ff8f204-tls-certs\" (UniqueName: \"kubernetes.io/secret/3360dc0f-497d-4211-a73f-c5ac0ff8f204-tls-certs\") pod \"3360dc0f-497d-4211-a73f-c5ac0ff8f204\" (UID: \"3360dc0f-497d-4211-a73f-c5ac0ff8f204\") " Oct 30 22:30:59.732588 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:59.732453 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3360dc0f-497d-4211-a73f-c5ac0ff8f204-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3360dc0f-497d-4211-a73f-c5ac0ff8f204-dshm\") pod \"3360dc0f-497d-4211-a73f-c5ac0ff8f204\" (UID: \"3360dc0f-497d-4211-a73f-c5ac0ff8f204\") " Oct 30 22:30:59.732588 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:59.732495 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3360dc0f-497d-4211-a73f-c5ac0ff8f204-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3360dc0f-497d-4211-a73f-c5ac0ff8f204-model-cache\") pod \"3360dc0f-497d-4211-a73f-c5ac0ff8f204\" (UID: \"3360dc0f-497d-4211-a73f-c5ac0ff8f204\") " Oct 30 22:30:59.732588 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:59.732498 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3360dc0f-497d-4211-a73f-c5ac0ff8f204-home" pod "3360dc0f-497d-4211-a73f-c5ac0ff8f204" (UID: "3360dc0f-497d-4211-a73f-c5ac0ff8f204"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:30:59.732830 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:59.732801 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3360dc0f-497d-4211-a73f-c5ac0ff8f204-model-cache" pod "3360dc0f-497d-4211-a73f-c5ac0ff8f204" (UID: "3360dc0f-497d-4211-a73f-c5ac0ff8f204"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:30:59.733133 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:59.733114 6720 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3360dc0f-497d-4211-a73f-c5ac0ff8f204-model-cache\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:59.733225 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:59.733138 6720 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3360dc0f-497d-4211-a73f-c5ac0ff8f204-tmp-dir\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:59.733225 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:59.733153 6720 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3360dc0f-497d-4211-a73f-c5ac0ff8f204-home\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:59.735185 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:59.735065 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3360dc0f-497d-4211-a73f-c5ac0ff8f204-tls-certs" pod "3360dc0f-497d-4211-a73f-c5ac0ff8f204" (UID: "3360dc0f-497d-4211-a73f-c5ac0ff8f204"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:30:59.735185 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:59.735153 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3360dc0f-497d-4211-a73f-c5ac0ff8f204-dshm" pod "3360dc0f-497d-4211-a73f-c5ac0ff8f204" (UID: "3360dc0f-497d-4211-a73f-c5ac0ff8f204"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:30:59.735352 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:59.735329 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3360dc0f-497d-4211-a73f-c5ac0ff8f204-kube-api-access-q98jl" pod "3360dc0f-497d-4211-a73f-c5ac0ff8f204" (UID: "3360dc0f-497d-4211-a73f-c5ac0ff8f204"). InnerVolumeSpecName "kube-api-access-q98jl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:59.835018 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:59.834932 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q98jl\" (UniqueName: \"kubernetes.io/projected/3360dc0f-497d-4211-a73f-c5ac0ff8f204-kube-api-access-q98jl\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:59.835018 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:59.834965 6720 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3360dc0f-497d-4211-a73f-c5ac0ff8f204-tls-certs\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:59.835018 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:59.834976 6720 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3360dc0f-497d-4211-a73f-c5ac0ff8f204-dshm\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:59.864226 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:30:59.864189 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:00.081410 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:00.062878 6720 generic.go:356] "Generic (PLEG): container finished" podID="3360dc0f-497d-4211-a73f-c5ac0ff8f204" containerID="b9cc50e667f7a065cc2c30ba3bc6625d8c292eb015208a9980eba973d6b9d88e" exitCode=0 Oct 30 22:31:00.081410 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:00.063021 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-df200a/test-model-df200a-kserve-7fc978d98f-jc487" event={"ID":"3360dc0f-497d-4211-a73f-c5ac0ff8f204","Type":"ContainerDied","Data":"b9cc50e667f7a065cc2c30ba3bc6625d8c292eb015208a9980eba973d6b9d88e"} Oct 30 22:31:00.081410 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:00.063153 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-df200a/test-model-df200a-kserve-7fc978d98f-jc487" event={"ID":"3360dc0f-497d-4211-a73f-c5ac0ff8f204","Type":"ContainerDied","Data":"a73425fb5d0dadbbc42e1cfa9308d8a1af27f77a3c267da29e54405f9e48a200"} Oct 30 22:31:00.081410 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:00.063180 6720 scope.go:122] "RemoveContainer" containerID="b9cc50e667f7a065cc2c30ba3bc6625d8c292eb015208a9980eba973d6b9d88e" Oct 30 22:31:00.081410 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:00.063917 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-df200a/test-model-df200a-kserve-7fc978d98f-jc487" Oct 30 22:31:00.094653 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:00.093908 6720 scope.go:122] "RemoveContainer" containerID="b9cc50e667f7a065cc2c30ba3bc6625d8c292eb015208a9980eba973d6b9d88e" Oct 30 22:31:00.094653 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:31:00.094276 6720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9cc50e667f7a065cc2c30ba3bc6625d8c292eb015208a9980eba973d6b9d88e\": container with ID starting with b9cc50e667f7a065cc2c30ba3bc6625d8c292eb015208a9980eba973d6b9d88e not found: ID does not exist" containerID="b9cc50e667f7a065cc2c30ba3bc6625d8c292eb015208a9980eba973d6b9d88e" Oct 30 22:31:00.094653 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:00.094343 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9cc50e667f7a065cc2c30ba3bc6625d8c292eb015208a9980eba973d6b9d88e"} err="failed to get container status \"b9cc50e667f7a065cc2c30ba3bc6625d8c292eb015208a9980eba973d6b9d88e\": rpc error: code = NotFound desc = could not find container \"b9cc50e667f7a065cc2c30ba3bc6625d8c292eb015208a9980eba973d6b9d88e\": container with ID starting with b9cc50e667f7a065cc2c30ba3bc6625d8c292eb015208a9980eba973d6b9d88e not found: ID does not exist" Oct 30 22:31:00.118863 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:00.118834 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-df200a/test-model-df200a-kserve-7fc978d98f-jc487"] Oct 30 22:31:00.127833 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:00.127443 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-df200a/test-model-df200a-kserve-7fc978d98f-jc487"] Oct 30 22:31:00.952410 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:00.932840 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:01.205644 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:01.204925 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:01.959269 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:01.959177 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3360dc0f-497d-4211-a73f-c5ac0ff8f204" path="/var/lib/kubelet/pods/3360dc0f-497d-4211-a73f-c5ac0ff8f204/volumes" Oct 30 22:31:02.351019 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:02.350922 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:03.250518 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:03.250483 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:04.036218 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:04.036166 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:04.348938 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:04.348865 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:04.547563 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:04.547520 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:04.953951 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:04.953915 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:05.884962 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:05.884928 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:06.563265 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:06.563227 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:07.393284 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:07.393243 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:08.303560 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:08.303528 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:08.789861 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:08.789807 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-58d5fa/test-model-58d5fa-kserve-746d5495f7-88wps" podUID="5ae33413-cc17-4bbb-ac91-ec406f02c125" containerName="main" containerID="cri-o://2fe82e1a7c0442818f6a40b742114d7597fc7ea14cb4303861fb5ea1e5d79bc3" gracePeriod=45 Oct 30 22:31:09.126237 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:09.126204 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-58d5fa/test-model-58d5fa-kserve-746d5495f7-88wps" Oct 30 22:31:09.129470 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:09.129404 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:09.206649 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:09.206611 6720 generic.go:356] "Generic (PLEG): container finished" podID="5ae33413-cc17-4bbb-ac91-ec406f02c125" containerID="2fe82e1a7c0442818f6a40b742114d7597fc7ea14cb4303861fb5ea1e5d79bc3" exitCode=0 Oct 30 22:31:09.206876 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:09.206670 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-58d5fa/test-model-58d5fa-kserve-746d5495f7-88wps" Oct 30 22:31:09.206876 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:09.206783 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-58d5fa/test-model-58d5fa-kserve-746d5495f7-88wps" event={"ID":"5ae33413-cc17-4bbb-ac91-ec406f02c125","Type":"ContainerDied","Data":"2fe82e1a7c0442818f6a40b742114d7597fc7ea14cb4303861fb5ea1e5d79bc3"} Oct 30 22:31:09.206876 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:09.206821 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-58d5fa/test-model-58d5fa-kserve-746d5495f7-88wps" event={"ID":"5ae33413-cc17-4bbb-ac91-ec406f02c125","Type":"ContainerDied","Data":"93529e161250d94e3e9c5b652f164e8b69545d942bbdc85a5c9c4d2e18414e5f"} Oct 30 22:31:09.206876 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:09.206843 6720 scope.go:122] "RemoveContainer" containerID="2fe82e1a7c0442818f6a40b742114d7597fc7ea14cb4303861fb5ea1e5d79bc3" Oct 30 22:31:09.218686 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:09.218666 6720 scope.go:122] "RemoveContainer" containerID="2fe82e1a7c0442818f6a40b742114d7597fc7ea14cb4303861fb5ea1e5d79bc3" Oct 30 22:31:09.218781 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:09.218708 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ae33413-cc17-4bbb-ac91-ec406f02c125-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ae33413-cc17-4bbb-ac91-ec406f02c125-tmp-dir\") pod \"5ae33413-cc17-4bbb-ac91-ec406f02c125\" (UID: \"5ae33413-cc17-4bbb-ac91-ec406f02c125\") " Oct 30 22:31:09.218837 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:09.218798 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ae33413-cc17-4bbb-ac91-ec406f02c125-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ae33413-cc17-4bbb-ac91-ec406f02c125-dshm\") pod \"5ae33413-cc17-4bbb-ac91-ec406f02c125\" (UID: \"5ae33413-cc17-4bbb-ac91-ec406f02c125\") " Oct 30 22:31:09.218932 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:09.218913 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ae33413-cc17-4bbb-ac91-ec406f02c125-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ae33413-cc17-4bbb-ac91-ec406f02c125-model-cache\") pod \"5ae33413-cc17-4bbb-ac91-ec406f02c125\" (UID: \"5ae33413-cc17-4bbb-ac91-ec406f02c125\") " Oct 30 22:31:09.218998 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:31:09.218949 6720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fe82e1a7c0442818f6a40b742114d7597fc7ea14cb4303861fb5ea1e5d79bc3\": container with ID starting with 2fe82e1a7c0442818f6a40b742114d7597fc7ea14cb4303861fb5ea1e5d79bc3 not found: ID does not exist" containerID="2fe82e1a7c0442818f6a40b742114d7597fc7ea14cb4303861fb5ea1e5d79bc3" Oct 30 22:31:09.218998 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:09.218978 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fe82e1a7c0442818f6a40b742114d7597fc7ea14cb4303861fb5ea1e5d79bc3"} err="failed to get container status \"2fe82e1a7c0442818f6a40b742114d7597fc7ea14cb4303861fb5ea1e5d79bc3\": rpc error: code = NotFound desc = could not find container \"2fe82e1a7c0442818f6a40b742114d7597fc7ea14cb4303861fb5ea1e5d79bc3\": container with ID starting with 2fe82e1a7c0442818f6a40b742114d7597fc7ea14cb4303861fb5ea1e5d79bc3 not found: ID does not exist" Oct 30 22:31:09.219152 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:09.219073 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ae33413-cc17-4bbb-ac91-ec406f02c125-kube-api-access-vdrtw\" (UniqueName: \"kubernetes.io/projected/5ae33413-cc17-4bbb-ac91-ec406f02c125-kube-api-access-vdrtw\") pod \"5ae33413-cc17-4bbb-ac91-ec406f02c125\" (UID: \"5ae33413-cc17-4bbb-ac91-ec406f02c125\") " Oct 30 22:31:09.219152 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:09.219099 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ae33413-cc17-4bbb-ac91-ec406f02c125-tmp-dir" pod "5ae33413-cc17-4bbb-ac91-ec406f02c125" (UID: "5ae33413-cc17-4bbb-ac91-ec406f02c125"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:09.219152 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:09.219137 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ae33413-cc17-4bbb-ac91-ec406f02c125-home\" (UniqueName: \"kubernetes.io/empty-dir/5ae33413-cc17-4bbb-ac91-ec406f02c125-home\") pod \"5ae33413-cc17-4bbb-ac91-ec406f02c125\" (UID: \"5ae33413-cc17-4bbb-ac91-ec406f02c125\") " Oct 30 22:31:09.219285 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:09.219224 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ae33413-cc17-4bbb-ac91-ec406f02c125-tls-certs\" (UniqueName: \"kubernetes.io/secret/5ae33413-cc17-4bbb-ac91-ec406f02c125-tls-certs\") pod \"5ae33413-cc17-4bbb-ac91-ec406f02c125\" (UID: \"5ae33413-cc17-4bbb-ac91-ec406f02c125\") " Oct 30 22:31:09.219336 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:09.219286 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ae33413-cc17-4bbb-ac91-ec406f02c125-model-cache" pod "5ae33413-cc17-4bbb-ac91-ec406f02c125" (UID: "5ae33413-cc17-4bbb-ac91-ec406f02c125"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:09.219463 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:09.219442 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ae33413-cc17-4bbb-ac91-ec406f02c125-home" pod "5ae33413-cc17-4bbb-ac91-ec406f02c125" (UID: "5ae33413-cc17-4bbb-ac91-ec406f02c125"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:09.219840 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:09.219820 6720 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ae33413-cc17-4bbb-ac91-ec406f02c125-model-cache\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:09.219961 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:09.219842 6720 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ae33413-cc17-4bbb-ac91-ec406f02c125-home\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:09.219961 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:09.219858 6720 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ae33413-cc17-4bbb-ac91-ec406f02c125-tmp-dir\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:09.221774 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:09.221751 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ae33413-cc17-4bbb-ac91-ec406f02c125-kube-api-access-vdrtw" pod "5ae33413-cc17-4bbb-ac91-ec406f02c125" (UID: "5ae33413-cc17-4bbb-ac91-ec406f02c125"). InnerVolumeSpecName "kube-api-access-vdrtw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:09.221894 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:09.221821 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ae33413-cc17-4bbb-ac91-ec406f02c125-tls-certs" pod "5ae33413-cc17-4bbb-ac91-ec406f02c125" (UID: "5ae33413-cc17-4bbb-ac91-ec406f02c125"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:31:09.221894 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:09.221825 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ae33413-cc17-4bbb-ac91-ec406f02c125-dshm" pod "5ae33413-cc17-4bbb-ac91-ec406f02c125" (UID: "5ae33413-cc17-4bbb-ac91-ec406f02c125"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:09.320900 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:09.320865 6720 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ae33413-cc17-4bbb-ac91-ec406f02c125-tls-certs\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:09.320900 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:09.320894 6720 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ae33413-cc17-4bbb-ac91-ec406f02c125-dshm\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:09.320900 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:09.320904 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdrtw\" (UniqueName: \"kubernetes.io/projected/5ae33413-cc17-4bbb-ac91-ec406f02c125-kube-api-access-vdrtw\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:09.536339 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:09.536306 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-58d5fa/test-model-58d5fa-kserve-746d5495f7-88wps"] Oct 30 22:31:09.546197 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:09.546165 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-58d5fa/test-model-58d5fa-kserve-746d5495f7-88wps"] Oct 30 22:31:09.922287 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:09.922240 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:09.967761 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:09.967724 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ae33413-cc17-4bbb-ac91-ec406f02c125" path="/var/lib/kubelet/pods/5ae33413-cc17-4bbb-ac91-ec406f02c125/volumes" Oct 30 22:31:10.404830 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:31:10.404796 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 22:31:10.668050 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:10.667956 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:11.395409 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:11.388905 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:11.846112 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:11.846010 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:12.013204 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:12.013155 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:12.227782 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:12.227740 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:12.952309 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:12.952271 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:13.818953 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:13.818917 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:14.555034 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:14.554993 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:15.377889 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:15.377848 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:15.540451 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:15.540416 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:16.616570 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:16.616532 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:17.818245 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:17.818205 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:18.537348 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:18.537312 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:19.354938 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:19.354905 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:20.327409 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:20.326647 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:21.084777 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:21.084742 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:21.641359 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:21.641318 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-28d39b-5b4c7f5c97-qsnhq"] Oct 30 22:31:21.641715 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:21.641688 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28d39b-5b4c7f5c97-qsnhq" podUID="a771bf56-fa60-4bac-bfd3-4aa0ccf2bf82" containerName="maas-api" containerID="cri-o://cff26aa9634b9da60583f367d5540bcaae964a2b369db37edf93176fd2877780" gracePeriod=30 Oct 30 22:31:21.739380 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:21.739345 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-7pjjd"] Oct 30 22:31:21.739684 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:21.739630 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-7pjjd" podUID="d31cf31e-58d7-4ace-8959-6cba1429daec" containerName="istio-proxy" containerID="cri-o://beb3f5fb7f2a4ad15bd674de0f8cd0926921c23ffd279c401cabbdd8c414999f" gracePeriod=30 Oct 30 22:31:21.753471 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:21.753183 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:22.027179 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:22.026221 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:22.187233 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:22.187199 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:22.216059 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:22.216032 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28d39b-5b4c7f5c97-qsnhq" Oct 30 22:31:22.338706 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:22.338615 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:22.363783 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:22.363751 6720 generic.go:356] "Generic (PLEG): container finished" podID="a771bf56-fa60-4bac-bfd3-4aa0ccf2bf82" containerID="cff26aa9634b9da60583f367d5540bcaae964a2b369db37edf93176fd2877780" exitCode=0 Oct 30 22:31:22.363968 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:22.363826 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28d39b-5b4c7f5c97-qsnhq" Oct 30 22:31:22.363968 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:22.363840 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28d39b-5b4c7f5c97-qsnhq" event={"ID":"a771bf56-fa60-4bac-bfd3-4aa0ccf2bf82","Type":"ContainerDied","Data":"cff26aa9634b9da60583f367d5540bcaae964a2b369db37edf93176fd2877780"} Oct 30 22:31:22.363968 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:22.363879 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28d39b-5b4c7f5c97-qsnhq" event={"ID":"a771bf56-fa60-4bac-bfd3-4aa0ccf2bf82","Type":"ContainerDied","Data":"162a7653a34d5148b8e1c19212a1672db18247424e714dac5565ff7835df8665"} Oct 30 22:31:22.363968 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:22.363903 6720 scope.go:122] "RemoveContainer" containerID="cff26aa9634b9da60583f367d5540bcaae964a2b369db37edf93176fd2877780" Oct 30 22:31:22.371877 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:22.371845 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a771bf56-fa60-4bac-bfd3-4aa0ccf2bf82-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a771bf56-fa60-4bac-bfd3-4aa0ccf2bf82-maas-api-tls\") pod \"a771bf56-fa60-4bac-bfd3-4aa0ccf2bf82\" (UID: \"a771bf56-fa60-4bac-bfd3-4aa0ccf2bf82\") " Oct 30 22:31:22.372033 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:22.371900 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a771bf56-fa60-4bac-bfd3-4aa0ccf2bf82-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a771bf56-fa60-4bac-bfd3-4aa0ccf2bf82-metrics-certs\") pod \"a771bf56-fa60-4bac-bfd3-4aa0ccf2bf82\" (UID: \"a771bf56-fa60-4bac-bfd3-4aa0ccf2bf82\") " Oct 30 22:31:22.372033 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:22.372023 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a771bf56-fa60-4bac-bfd3-4aa0ccf2bf82-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a771bf56-fa60-4bac-bfd3-4aa0ccf2bf82-openshift-service-ca\") pod \"a771bf56-fa60-4bac-bfd3-4aa0ccf2bf82\" (UID: \"a771bf56-fa60-4bac-bfd3-4aa0ccf2bf82\") " Oct 30 22:31:22.372203 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:22.372193 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a771bf56-fa60-4bac-bfd3-4aa0ccf2bf82-kube-api-access-4j4lm\" (UniqueName: \"kubernetes.io/projected/a771bf56-fa60-4bac-bfd3-4aa0ccf2bf82-kube-api-access-4j4lm\") pod \"a771bf56-fa60-4bac-bfd3-4aa0ccf2bf82\" (UID: \"a771bf56-fa60-4bac-bfd3-4aa0ccf2bf82\") " Oct 30 22:31:22.373102 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:22.372890 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a771bf56-fa60-4bac-bfd3-4aa0ccf2bf82-openshift-service-ca" pod "a771bf56-fa60-4bac-bfd3-4aa0ccf2bf82" (UID: "a771bf56-fa60-4bac-bfd3-4aa0ccf2bf82"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:31:22.378792 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:22.376512 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a771bf56-fa60-4bac-bfd3-4aa0ccf2bf82-kube-api-access-4j4lm" pod "a771bf56-fa60-4bac-bfd3-4aa0ccf2bf82" (UID: "a771bf56-fa60-4bac-bfd3-4aa0ccf2bf82"). InnerVolumeSpecName "kube-api-access-4j4lm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:22.378792 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:22.378724 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a771bf56-fa60-4bac-bfd3-4aa0ccf2bf82-maas-api-tls" pod "a771bf56-fa60-4bac-bfd3-4aa0ccf2bf82" (UID: "a771bf56-fa60-4bac-bfd3-4aa0ccf2bf82"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:31:22.380114 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:22.380089 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a771bf56-fa60-4bac-bfd3-4aa0ccf2bf82-metrics-certs" pod "a771bf56-fa60-4bac-bfd3-4aa0ccf2bf82" (UID: "a771bf56-fa60-4bac-bfd3-4aa0ccf2bf82"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:31:22.384374 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:22.384350 6720 scope.go:122] "RemoveContainer" containerID="cff26aa9634b9da60583f367d5540bcaae964a2b369db37edf93176fd2877780" Oct 30 22:31:22.385313 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:31:22.385280 6720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cff26aa9634b9da60583f367d5540bcaae964a2b369db37edf93176fd2877780\": container with ID starting with cff26aa9634b9da60583f367d5540bcaae964a2b369db37edf93176fd2877780 not found: ID does not exist" containerID="cff26aa9634b9da60583f367d5540bcaae964a2b369db37edf93176fd2877780" Oct 30 22:31:22.385437 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:22.385323 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cff26aa9634b9da60583f367d5540bcaae964a2b369db37edf93176fd2877780"} err="failed to get container status \"cff26aa9634b9da60583f367d5540bcaae964a2b369db37edf93176fd2877780\": rpc error: code = NotFound desc = could not find container \"cff26aa9634b9da60583f367d5540bcaae964a2b369db37edf93176fd2877780\": container with ID starting with cff26aa9634b9da60583f367d5540bcaae964a2b369db37edf93176fd2877780 not found: ID does not exist" Oct 30 22:31:22.475099 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:22.475063 6720 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a771bf56-fa60-4bac-bfd3-4aa0ccf2bf82-maas-api-tls\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:22.475099 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:22.475097 6720 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a771bf56-fa60-4bac-bfd3-4aa0ccf2bf82-metrics-certs\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:22.475269 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:22.475107 6720 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a771bf56-fa60-4bac-bfd3-4aa0ccf2bf82-openshift-service-ca\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:22.475269 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:22.475117 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4j4lm\" (UniqueName: \"kubernetes.io/projected/a771bf56-fa60-4bac-bfd3-4aa0ccf2bf82-kube-api-access-4j4lm\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:22.721179 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:22.721141 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-28d39b-5b4c7f5c97-qsnhq"] Oct 30 22:31:22.741842 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:22.741815 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-28d39b-5b4c7f5c97-qsnhq"] Oct 30 22:31:22.828679 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:22.828641 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:23.313796 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:23.313760 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:23.807414 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:23.807029 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-df200a-7684bb76b6-rjq5j"] Oct 30 22:31:23.807414 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:23.807264 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df200a-7684bb76b6-rjq5j" podUID="1f53144d-ae7f-493c-ac5b-714b9ad3b29e" containerName="maas-api" containerID="cri-o://9f0e80e6c01b1807d237a38be40889f98212dc1320b69e02c6884215a5643fc5" gracePeriod=30 Oct 30 22:31:23.907892 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:23.907859 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:23.961879 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:23.961847 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a771bf56-fa60-4bac-bfd3-4aa0ccf2bf82" path="/var/lib/kubelet/pods/a771bf56-fa60-4bac-bfd3-4aa0ccf2bf82/volumes" Oct 30 22:31:24.314048 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:24.313998 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:24.372111 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:24.372083 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df200a-7684bb76b6-rjq5j" Oct 30 22:31:24.386419 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:24.385472 6720 generic.go:356] "Generic (PLEG): container finished" podID="1f53144d-ae7f-493c-ac5b-714b9ad3b29e" containerID="9f0e80e6c01b1807d237a38be40889f98212dc1320b69e02c6884215a5643fc5" exitCode=0 Oct 30 22:31:24.386419 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:24.385694 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df200a-7684bb76b6-rjq5j" event={"ID":"1f53144d-ae7f-493c-ac5b-714b9ad3b29e","Type":"ContainerDied","Data":"9f0e80e6c01b1807d237a38be40889f98212dc1320b69e02c6884215a5643fc5"} Oct 30 22:31:24.386419 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:24.385753 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df200a-7684bb76b6-rjq5j" event={"ID":"1f53144d-ae7f-493c-ac5b-714b9ad3b29e","Type":"ContainerDied","Data":"95fe21c7b7404ab25dc9017d3af9d3667ad290a8eac1c0acaf6d6d15af6aee75"} Oct 30 22:31:24.386419 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:24.385788 6720 scope.go:122] "RemoveContainer" containerID="9f0e80e6c01b1807d237a38be40889f98212dc1320b69e02c6884215a5643fc5" Oct 30 22:31:24.386419 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:24.386189 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-df200a-7684bb76b6-rjq5j" Oct 30 22:31:24.435551 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:24.435492 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f53144d-ae7f-493c-ac5b-714b9ad3b29e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f53144d-ae7f-493c-ac5b-714b9ad3b29e-metrics-certs\") pod \"1f53144d-ae7f-493c-ac5b-714b9ad3b29e\" (UID: \"1f53144d-ae7f-493c-ac5b-714b9ad3b29e\") " Oct 30 22:31:24.435854 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:24.435814 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f53144d-ae7f-493c-ac5b-714b9ad3b29e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f53144d-ae7f-493c-ac5b-714b9ad3b29e-openshift-service-ca\") pod \"1f53144d-ae7f-493c-ac5b-714b9ad3b29e\" (UID: \"1f53144d-ae7f-493c-ac5b-714b9ad3b29e\") " Oct 30 22:31:24.435979 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:24.435898 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f53144d-ae7f-493c-ac5b-714b9ad3b29e-kube-api-access-rrnqw\" (UniqueName: \"kubernetes.io/projected/1f53144d-ae7f-493c-ac5b-714b9ad3b29e-kube-api-access-rrnqw\") pod \"1f53144d-ae7f-493c-ac5b-714b9ad3b29e\" (UID: \"1f53144d-ae7f-493c-ac5b-714b9ad3b29e\") " Oct 30 22:31:24.436140 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:24.436121 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f53144d-ae7f-493c-ac5b-714b9ad3b29e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f53144d-ae7f-493c-ac5b-714b9ad3b29e-maas-api-tls\") pod \"1f53144d-ae7f-493c-ac5b-714b9ad3b29e\" (UID: \"1f53144d-ae7f-493c-ac5b-714b9ad3b29e\") " Oct 30 22:31:24.442855 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:24.442827 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f53144d-ae7f-493c-ac5b-714b9ad3b29e-maas-api-tls" pod "1f53144d-ae7f-493c-ac5b-714b9ad3b29e" (UID: "1f53144d-ae7f-493c-ac5b-714b9ad3b29e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:31:24.444943 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:24.444921 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f53144d-ae7f-493c-ac5b-714b9ad3b29e-metrics-certs" pod "1f53144d-ae7f-493c-ac5b-714b9ad3b29e" (UID: "1f53144d-ae7f-493c-ac5b-714b9ad3b29e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:31:24.445189 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:24.445174 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f53144d-ae7f-493c-ac5b-714b9ad3b29e-openshift-service-ca" pod "1f53144d-ae7f-493c-ac5b-714b9ad3b29e" (UID: "1f53144d-ae7f-493c-ac5b-714b9ad3b29e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:31:24.447577 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:24.447551 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f53144d-ae7f-493c-ac5b-714b9ad3b29e-kube-api-access-rrnqw" pod "1f53144d-ae7f-493c-ac5b-714b9ad3b29e" (UID: "1f53144d-ae7f-493c-ac5b-714b9ad3b29e"). InnerVolumeSpecName "kube-api-access-rrnqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:24.447720 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:24.447702 6720 scope.go:122] "RemoveContainer" containerID="9f0e80e6c01b1807d237a38be40889f98212dc1320b69e02c6884215a5643fc5" Oct 30 22:31:24.448243 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:31:24.448219 6720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f0e80e6c01b1807d237a38be40889f98212dc1320b69e02c6884215a5643fc5\": container with ID starting with 9f0e80e6c01b1807d237a38be40889f98212dc1320b69e02c6884215a5643fc5 not found: ID does not exist" containerID="9f0e80e6c01b1807d237a38be40889f98212dc1320b69e02c6884215a5643fc5" Oct 30 22:31:24.448288 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:24.448256 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f0e80e6c01b1807d237a38be40889f98212dc1320b69e02c6884215a5643fc5"} err="failed to get container status \"9f0e80e6c01b1807d237a38be40889f98212dc1320b69e02c6884215a5643fc5\": rpc error: code = NotFound desc = could not find container \"9f0e80e6c01b1807d237a38be40889f98212dc1320b69e02c6884215a5643fc5\": container with ID starting with 9f0e80e6c01b1807d237a38be40889f98212dc1320b69e02c6884215a5643fc5 not found: ID does not exist" Oct 30 22:31:24.468162 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:24.468139 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:24.538508 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:24.538476 6720 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f53144d-ae7f-493c-ac5b-714b9ad3b29e-metrics-certs\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:24.538508 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:24.538505 6720 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f53144d-ae7f-493c-ac5b-714b9ad3b29e-openshift-service-ca\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:24.538508 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:24.538518 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrnqw\" (UniqueName: \"kubernetes.io/projected/1f53144d-ae7f-493c-ac5b-714b9ad3b29e-kube-api-access-rrnqw\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:24.538762 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:24.538527 6720 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f53144d-ae7f-493c-ac5b-714b9ad3b29e-maas-api-tls\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:24.717334 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:24.717301 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-df200a-7684bb76b6-rjq5j"] Oct 30 22:31:24.723339 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:24.723310 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-df200a-7684bb76b6-rjq5j"] Oct 30 22:31:24.895768 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:24.895725 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:25.228097 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:25.228062 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:25.946997 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:25.945511 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:25.970495 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:25.970456 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f53144d-ae7f-493c-ac5b-714b9ad3b29e" path="/var/lib/kubelet/pods/1f53144d-ae7f-493c-ac5b-714b9ad3b29e/volumes" Oct 30 22:31:26.508466 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:26.508431 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-4b2bb0/test-model-4b2bb0-kserve-5df44c7bc5-mj9p7"] Oct 30 22:31:26.588083 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:26.588047 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:26.755306 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:26.755222 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:27.411923 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:27.411891 6720 generic.go:356] "Generic (PLEG): container finished" podID="d31cf31e-58d7-4ace-8959-6cba1429daec" containerID="beb3f5fb7f2a4ad15bd674de0f8cd0926921c23ffd279c401cabbdd8c414999f" exitCode=0 Oct 30 22:31:27.412300 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:27.411972 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-7pjjd" event={"ID":"d31cf31e-58d7-4ace-8959-6cba1429daec","Type":"ContainerDied","Data":"beb3f5fb7f2a4ad15bd674de0f8cd0926921c23ffd279c401cabbdd8c414999f"} Oct 30 22:31:27.455646 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:27.455617 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-7pjjd" Oct 30 22:31:27.506010 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:27.505973 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d31cf31e-58d7-4ace-8959-6cba1429daec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d31cf31e-58d7-4ace-8959-6cba1429daec-workload-socket\") pod \"d31cf31e-58d7-4ace-8959-6cba1429daec\" (UID: \"d31cf31e-58d7-4ace-8959-6cba1429daec\") " Oct 30 22:31:27.506207 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:27.506033 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d31cf31e-58d7-4ace-8959-6cba1429daec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d31cf31e-58d7-4ace-8959-6cba1429daec-credential-socket\") pod \"d31cf31e-58d7-4ace-8959-6cba1429daec\" (UID: \"d31cf31e-58d7-4ace-8959-6cba1429daec\") " Oct 30 22:31:27.506207 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:27.506060 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d31cf31e-58d7-4ace-8959-6cba1429daec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d31cf31e-58d7-4ace-8959-6cba1429daec-istio-envoy\") pod \"d31cf31e-58d7-4ace-8959-6cba1429daec\" (UID: \"d31cf31e-58d7-4ace-8959-6cba1429daec\") " Oct 30 22:31:27.506207 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:27.506092 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d31cf31e-58d7-4ace-8959-6cba1429daec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d31cf31e-58d7-4ace-8959-6cba1429daec-istio-podinfo\") pod \"d31cf31e-58d7-4ace-8959-6cba1429daec\" (UID: \"d31cf31e-58d7-4ace-8959-6cba1429daec\") " Oct 30 22:31:27.506207 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:27.506150 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d31cf31e-58d7-4ace-8959-6cba1429daec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d31cf31e-58d7-4ace-8959-6cba1429daec-istiod-ca-cert\") pod \"d31cf31e-58d7-4ace-8959-6cba1429daec\" (UID: \"d31cf31e-58d7-4ace-8959-6cba1429daec\") " Oct 30 22:31:27.506468 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:27.506220 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d31cf31e-58d7-4ace-8959-6cba1429daec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d31cf31e-58d7-4ace-8959-6cba1429daec-istio-data\") pod \"d31cf31e-58d7-4ace-8959-6cba1429daec\" (UID: \"d31cf31e-58d7-4ace-8959-6cba1429daec\") " Oct 30 22:31:27.506468 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:27.506427 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d31cf31e-58d7-4ace-8959-6cba1429daec-istio-token\" (UniqueName: \"kubernetes.io/projected/d31cf31e-58d7-4ace-8959-6cba1429daec-istio-token\") pod \"d31cf31e-58d7-4ace-8959-6cba1429daec\" (UID: \"d31cf31e-58d7-4ace-8959-6cba1429daec\") " Oct 30 22:31:27.506576 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:27.506476 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d31cf31e-58d7-4ace-8959-6cba1429daec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d31cf31e-58d7-4ace-8959-6cba1429daec-workload-certs\") pod \"d31cf31e-58d7-4ace-8959-6cba1429daec\" (UID: \"d31cf31e-58d7-4ace-8959-6cba1429daec\") " Oct 30 22:31:27.506576 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:27.506552 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d31cf31e-58d7-4ace-8959-6cba1429daec-kube-api-access-72ptn\" (UniqueName: \"kubernetes.io/projected/d31cf31e-58d7-4ace-8959-6cba1429daec-kube-api-access-72ptn\") pod \"d31cf31e-58d7-4ace-8959-6cba1429daec\" (UID: \"d31cf31e-58d7-4ace-8959-6cba1429daec\") " Oct 30 22:31:27.508070 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:27.507902 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d31cf31e-58d7-4ace-8959-6cba1429daec-credential-socket" pod "d31cf31e-58d7-4ace-8959-6cba1429daec" (UID: "d31cf31e-58d7-4ace-8959-6cba1429daec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:27.508423 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:27.508227 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d31cf31e-58d7-4ace-8959-6cba1429daec-workload-socket" pod "d31cf31e-58d7-4ace-8959-6cba1429daec" (UID: "d31cf31e-58d7-4ace-8959-6cba1429daec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:27.508665 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:27.508615 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d31cf31e-58d7-4ace-8959-6cba1429daec-istio-data" pod "d31cf31e-58d7-4ace-8959-6cba1429daec" (UID: "d31cf31e-58d7-4ace-8959-6cba1429daec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:27.509227 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:27.509194 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d31cf31e-58d7-4ace-8959-6cba1429daec-workload-certs" pod "d31cf31e-58d7-4ace-8959-6cba1429daec" (UID: "d31cf31e-58d7-4ace-8959-6cba1429daec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:27.509720 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:27.509701 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d31cf31e-58d7-4ace-8959-6cba1429daec-istiod-ca-cert" pod "d31cf31e-58d7-4ace-8959-6cba1429daec" (UID: "d31cf31e-58d7-4ace-8959-6cba1429daec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:31:27.516983 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:27.516946 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d31cf31e-58d7-4ace-8959-6cba1429daec-kube-api-access-72ptn" pod "d31cf31e-58d7-4ace-8959-6cba1429daec" (UID: "d31cf31e-58d7-4ace-8959-6cba1429daec"). InnerVolumeSpecName "kube-api-access-72ptn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:27.517565 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:27.517538 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d31cf31e-58d7-4ace-8959-6cba1429daec-istio-podinfo" pod "d31cf31e-58d7-4ace-8959-6cba1429daec" (UID: "d31cf31e-58d7-4ace-8959-6cba1429daec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:31:27.517970 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:27.517942 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d31cf31e-58d7-4ace-8959-6cba1429daec-istio-token" pod "d31cf31e-58d7-4ace-8959-6cba1429daec" (UID: "d31cf31e-58d7-4ace-8959-6cba1429daec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:27.520801 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:27.520775 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d31cf31e-58d7-4ace-8959-6cba1429daec-istio-envoy" pod "d31cf31e-58d7-4ace-8959-6cba1429daec" (UID: "d31cf31e-58d7-4ace-8959-6cba1429daec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:27.609018 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:27.608983 6720 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d31cf31e-58d7-4ace-8959-6cba1429daec-istio-data\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:27.609018 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:27.609015 6720 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d31cf31e-58d7-4ace-8959-6cba1429daec-istio-token\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:27.609018 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:27.609024 6720 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d31cf31e-58d7-4ace-8959-6cba1429daec-workload-certs\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:27.609246 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:27.609034 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72ptn\" (UniqueName: \"kubernetes.io/projected/d31cf31e-58d7-4ace-8959-6cba1429daec-kube-api-access-72ptn\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:27.609246 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:27.609044 6720 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d31cf31e-58d7-4ace-8959-6cba1429daec-workload-socket\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:27.609246 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:27.609054 6720 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d31cf31e-58d7-4ace-8959-6cba1429daec-credential-socket\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:27.609246 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:27.609063 6720 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d31cf31e-58d7-4ace-8959-6cba1429daec-istio-envoy\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:27.609246 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:27.609070 6720 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d31cf31e-58d7-4ace-8959-6cba1429daec-istio-podinfo\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:27.609246 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:27.609078 6720 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d31cf31e-58d7-4ace-8959-6cba1429daec-istiod-ca-cert\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:27.873034 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:27.870553 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:28.421530 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:28.421503 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-7pjjd" Oct 30 22:31:28.422030 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:28.421498 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-7pjjd" event={"ID":"d31cf31e-58d7-4ace-8959-6cba1429daec","Type":"ContainerDied","Data":"747f0719d0615579cbe69cbe39dd43f10e9cd98eeb98cd4f354b515fe2cedcc5"} Oct 30 22:31:28.422030 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:28.421641 6720 scope.go:122] "RemoveContainer" containerID="beb3f5fb7f2a4ad15bd674de0f8cd0926921c23ffd279c401cabbdd8c414999f" Oct 30 22:31:28.481982 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:28.481952 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-7pjjd"] Oct 30 22:31:28.491930 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:28.491902 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-28b4bb-openshift-default-588879f6bf-7pjjd"] Oct 30 22:31:28.735609 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:28.735525 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:29.543483 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:29.539123 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:29.959495 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:29.959461 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d31cf31e-58d7-4ace-8959-6cba1429daec" path="/var/lib/kubelet/pods/d31cf31e-58d7-4ace-8959-6cba1429daec/volumes" Oct 30 22:31:30.217211 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:30.217114 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:30.476297 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:30.476206 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:31.139144 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:31.139080 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:31.317414 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:31.311604 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:31.393371 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:31.392493 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-vvpgw"] Oct 30 22:31:31.393371 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:31.392725 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-vvpgw" podUID="1ea7cd43-6136-4a44-921d-65fbf8c39f76" containerName="istio-proxy" containerID="cri-o://98f6ee11f0ee603c47f6dae9e5a65e88b1d9906f5412f67425e2e9980f6957c2" gracePeriod=30 Oct 30 22:31:31.516711 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:31.516665 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:32.220293 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:32.220259 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:32.520248 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:32.520134 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:32.784016 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:32.783931 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:32.939459 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:32.938719 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-xbbpj"] Oct 30 22:31:32.940027 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:32.939991 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-xbbpj" podUID="47d2f169-c1d8-4533-be16-9d195a7f73c9" containerName="istio-proxy" containerID="cri-o://713c79240f39a70ec0bbe2856a82666fd7cab78db9cfd6cbc110498580f9a2c8" gracePeriod=30 Oct 30 22:31:33.070474 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:33.070346 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:33.648021 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:33.647987 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:34.568980 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:34.568944 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:35.456316 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:35.456275 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:36.187272 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:36.187223 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:36.333521 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:36.333477 6720 patch_prober.go:29] interesting pod/e2e-inf-a-df200a-openshift-default-86d8f88f8f-xbbpj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:31:36.333770 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:36.333558 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-xbbpj" podUID="47d2f169-c1d8-4533-be16-9d195a7f73c9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:31:36.710459 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:36.710421 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-vvpgw" Oct 30 22:31:36.800471 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:36.800355 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1ea7cd43-6136-4a44-921d-65fbf8c39f76-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ea7cd43-6136-4a44-921d-65fbf8c39f76-istio-podinfo\") pod \"1ea7cd43-6136-4a44-921d-65fbf8c39f76\" (UID: \"1ea7cd43-6136-4a44-921d-65fbf8c39f76\") " Oct 30 22:31:36.800651 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:36.800469 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ea7cd43-6136-4a44-921d-65fbf8c39f76-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ea7cd43-6136-4a44-921d-65fbf8c39f76-istio-envoy\") pod \"1ea7cd43-6136-4a44-921d-65fbf8c39f76\" (UID: \"1ea7cd43-6136-4a44-921d-65fbf8c39f76\") " Oct 30 22:31:36.800651 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:36.800599 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ea7cd43-6136-4a44-921d-65fbf8c39f76-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ea7cd43-6136-4a44-921d-65fbf8c39f76-workload-socket\") pod \"1ea7cd43-6136-4a44-921d-65fbf8c39f76\" (UID: \"1ea7cd43-6136-4a44-921d-65fbf8c39f76\") " Oct 30 22:31:36.800651 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:36.800626 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ea7cd43-6136-4a44-921d-65fbf8c39f76-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ea7cd43-6136-4a44-921d-65fbf8c39f76-istiod-ca-cert\") pod \"1ea7cd43-6136-4a44-921d-65fbf8c39f76\" (UID: \"1ea7cd43-6136-4a44-921d-65fbf8c39f76\") " Oct 30 22:31:36.800834 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:36.800657 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ea7cd43-6136-4a44-921d-65fbf8c39f76-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ea7cd43-6136-4a44-921d-65fbf8c39f76-workload-certs\") pod \"1ea7cd43-6136-4a44-921d-65fbf8c39f76\" (UID: \"1ea7cd43-6136-4a44-921d-65fbf8c39f76\") " Oct 30 22:31:36.800834 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:36.800685 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ea7cd43-6136-4a44-921d-65fbf8c39f76-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ea7cd43-6136-4a44-921d-65fbf8c39f76-istio-data\") pod \"1ea7cd43-6136-4a44-921d-65fbf8c39f76\" (UID: \"1ea7cd43-6136-4a44-921d-65fbf8c39f76\") " Oct 30 22:31:36.800986 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:36.800894 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ea7cd43-6136-4a44-921d-65fbf8c39f76-workload-socket" pod "1ea7cd43-6136-4a44-921d-65fbf8c39f76" (UID: "1ea7cd43-6136-4a44-921d-65fbf8c39f76"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:36.800986 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:36.800912 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ea7cd43-6136-4a44-921d-65fbf8c39f76-kube-api-access-sq2n2\" (UniqueName: \"kubernetes.io/projected/1ea7cd43-6136-4a44-921d-65fbf8c39f76-kube-api-access-sq2n2\") pod \"1ea7cd43-6136-4a44-921d-65fbf8c39f76\" (UID: \"1ea7cd43-6136-4a44-921d-65fbf8c39f76\") " Oct 30 22:31:36.801110 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:36.801016 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ea7cd43-6136-4a44-921d-65fbf8c39f76-istio-data" pod "1ea7cd43-6136-4a44-921d-65fbf8c39f76" (UID: "1ea7cd43-6136-4a44-921d-65fbf8c39f76"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:36.801110 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:36.801022 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ea7cd43-6136-4a44-921d-65fbf8c39f76-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ea7cd43-6136-4a44-921d-65fbf8c39f76-credential-socket\") pod \"1ea7cd43-6136-4a44-921d-65fbf8c39f76\" (UID: \"1ea7cd43-6136-4a44-921d-65fbf8c39f76\") " Oct 30 22:31:36.801225 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:36.801118 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ea7cd43-6136-4a44-921d-65fbf8c39f76-istio-token\" (UniqueName: \"kubernetes.io/projected/1ea7cd43-6136-4a44-921d-65fbf8c39f76-istio-token\") pod \"1ea7cd43-6136-4a44-921d-65fbf8c39f76\" (UID: \"1ea7cd43-6136-4a44-921d-65fbf8c39f76\") " Oct 30 22:31:36.801225 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:36.801204 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ea7cd43-6136-4a44-921d-65fbf8c39f76-credential-socket" pod "1ea7cd43-6136-4a44-921d-65fbf8c39f76" (UID: "1ea7cd43-6136-4a44-921d-65fbf8c39f76"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:36.801522 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:36.801246 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ea7cd43-6136-4a44-921d-65fbf8c39f76-istiod-ca-cert" pod "1ea7cd43-6136-4a44-921d-65fbf8c39f76" (UID: "1ea7cd43-6136-4a44-921d-65fbf8c39f76"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:31:36.801585 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:36.801528 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ea7cd43-6136-4a44-921d-65fbf8c39f76-workload-certs" pod "1ea7cd43-6136-4a44-921d-65fbf8c39f76" (UID: "1ea7cd43-6136-4a44-921d-65fbf8c39f76"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:36.802099 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:36.801841 6720 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ea7cd43-6136-4a44-921d-65fbf8c39f76-workload-socket\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:36.802099 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:36.801866 6720 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ea7cd43-6136-4a44-921d-65fbf8c39f76-istiod-ca-cert\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:36.802099 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:36.801881 6720 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ea7cd43-6136-4a44-921d-65fbf8c39f76-workload-certs\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:36.802099 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:36.801900 6720 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ea7cd43-6136-4a44-921d-65fbf8c39f76-istio-data\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:36.802099 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:36.801914 6720 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ea7cd43-6136-4a44-921d-65fbf8c39f76-credential-socket\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:36.803720 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:36.803669 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ea7cd43-6136-4a44-921d-65fbf8c39f76-istio-podinfo" pod "1ea7cd43-6136-4a44-921d-65fbf8c39f76" (UID: "1ea7cd43-6136-4a44-921d-65fbf8c39f76"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:31:36.804140 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:36.804102 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ea7cd43-6136-4a44-921d-65fbf8c39f76-kube-api-access-sq2n2" pod "1ea7cd43-6136-4a44-921d-65fbf8c39f76" (UID: "1ea7cd43-6136-4a44-921d-65fbf8c39f76"). InnerVolumeSpecName "kube-api-access-sq2n2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:36.804271 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:36.804204 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ea7cd43-6136-4a44-921d-65fbf8c39f76-istio-envoy" pod "1ea7cd43-6136-4a44-921d-65fbf8c39f76" (UID: "1ea7cd43-6136-4a44-921d-65fbf8c39f76"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:36.804856 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:36.804813 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ea7cd43-6136-4a44-921d-65fbf8c39f76-istio-token" pod "1ea7cd43-6136-4a44-921d-65fbf8c39f76" (UID: "1ea7cd43-6136-4a44-921d-65fbf8c39f76"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:36.841112 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:36.841069 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:36.903379 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:36.903346 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sq2n2\" (UniqueName: \"kubernetes.io/projected/1ea7cd43-6136-4a44-921d-65fbf8c39f76-kube-api-access-sq2n2\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:36.903379 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:36.903377 6720 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ea7cd43-6136-4a44-921d-65fbf8c39f76-istio-token\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:36.903641 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:36.903437 6720 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ea7cd43-6136-4a44-921d-65fbf8c39f76-istio-podinfo\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:36.903641 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:36.903447 6720 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ea7cd43-6136-4a44-921d-65fbf8c39f76-istio-envoy\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:37.031347 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:37.031317 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:37.215415 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:37.201606 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:37.500259 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:37.500170 6720 generic.go:356] "Generic (PLEG): container finished" podID="1ea7cd43-6136-4a44-921d-65fbf8c39f76" containerID="98f6ee11f0ee603c47f6dae9e5a65e88b1d9906f5412f67425e2e9980f6957c2" exitCode=0 Oct 30 22:31:37.500259 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:37.500224 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-vvpgw" Oct 30 22:31:37.500501 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:37.500223 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-vvpgw" event={"ID":"1ea7cd43-6136-4a44-921d-65fbf8c39f76","Type":"ContainerDied","Data":"98f6ee11f0ee603c47f6dae9e5a65e88b1d9906f5412f67425e2e9980f6957c2"} Oct 30 22:31:37.500501 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:37.500363 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-vvpgw" event={"ID":"1ea7cd43-6136-4a44-921d-65fbf8c39f76","Type":"ContainerDied","Data":"863ed938305bce8862cc7c55e0487a38cc328a1c511c0ae0ce1a35c9109b7c10"} Oct 30 22:31:37.500501 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:37.500411 6720 scope.go:122] "RemoveContainer" containerID="98f6ee11f0ee603c47f6dae9e5a65e88b1d9906f5412f67425e2e9980f6957c2" Oct 30 22:31:37.512194 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:37.512179 6720 scope.go:122] "RemoveContainer" containerID="98f6ee11f0ee603c47f6dae9e5a65e88b1d9906f5412f67425e2e9980f6957c2" Oct 30 22:31:37.512510 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:31:37.512482 6720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98f6ee11f0ee603c47f6dae9e5a65e88b1d9906f5412f67425e2e9980f6957c2\": container with ID starting with 98f6ee11f0ee603c47f6dae9e5a65e88b1d9906f5412f67425e2e9980f6957c2 not found: ID does not exist" containerID="98f6ee11f0ee603c47f6dae9e5a65e88b1d9906f5412f67425e2e9980f6957c2" Oct 30 22:31:37.512610 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:37.512516 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98f6ee11f0ee603c47f6dae9e5a65e88b1d9906f5412f67425e2e9980f6957c2"} err="failed to get container status \"98f6ee11f0ee603c47f6dae9e5a65e88b1d9906f5412f67425e2e9980f6957c2\": rpc error: code = NotFound desc = could not find container \"98f6ee11f0ee603c47f6dae9e5a65e88b1d9906f5412f67425e2e9980f6957c2\": container with ID starting with 98f6ee11f0ee603c47f6dae9e5a65e88b1d9906f5412f67425e2e9980f6957c2 not found: ID does not exist" Oct 30 22:31:37.529633 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:37.529610 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-vvpgw"] Oct 30 22:31:37.533477 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:37.533454 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-28d39b-openshift-default-5544d7c55f-vvpgw"] Oct 30 22:31:37.628323 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:37.628286 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:37.957715 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:37.957685 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ea7cd43-6136-4a44-921d-65fbf8c39f76" path="/var/lib/kubelet/pods/1ea7cd43-6136-4a44-921d-65fbf8c39f76/volumes" Oct 30 22:31:38.164329 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:38.164302 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-xbbpj" Oct 30 22:31:38.233802 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:38.233715 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47d2f169-c1d8-4533-be16-9d195a7f73c9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d2f169-c1d8-4533-be16-9d195a7f73c9-credential-socket\") pod \"47d2f169-c1d8-4533-be16-9d195a7f73c9\" (UID: \"47d2f169-c1d8-4533-be16-9d195a7f73c9\") " Oct 30 22:31:38.233802 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:38.233776 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47d2f169-c1d8-4533-be16-9d195a7f73c9-kube-api-access-626xp\" (UniqueName: \"kubernetes.io/projected/47d2f169-c1d8-4533-be16-9d195a7f73c9-kube-api-access-626xp\") pod \"47d2f169-c1d8-4533-be16-9d195a7f73c9\" (UID: \"47d2f169-c1d8-4533-be16-9d195a7f73c9\") " Oct 30 22:31:38.234038 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:38.233830 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/47d2f169-c1d8-4533-be16-9d195a7f73c9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47d2f169-c1d8-4533-be16-9d195a7f73c9-istio-podinfo\") pod \"47d2f169-c1d8-4533-be16-9d195a7f73c9\" (UID: \"47d2f169-c1d8-4533-be16-9d195a7f73c9\") " Oct 30 22:31:38.234038 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:38.233865 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47d2f169-c1d8-4533-be16-9d195a7f73c9-istio-token\" (UniqueName: \"kubernetes.io/projected/47d2f169-c1d8-4533-be16-9d195a7f73c9-istio-token\") pod \"47d2f169-c1d8-4533-be16-9d195a7f73c9\" (UID: \"47d2f169-c1d8-4533-be16-9d195a7f73c9\") " Oct 30 22:31:38.234038 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:38.233910 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47d2f169-c1d8-4533-be16-9d195a7f73c9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d2f169-c1d8-4533-be16-9d195a7f73c9-workload-socket\") pod \"47d2f169-c1d8-4533-be16-9d195a7f73c9\" (UID: \"47d2f169-c1d8-4533-be16-9d195a7f73c9\") " Oct 30 22:31:38.234038 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:38.233944 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47d2f169-c1d8-4533-be16-9d195a7f73c9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47d2f169-c1d8-4533-be16-9d195a7f73c9-istio-data\") pod \"47d2f169-c1d8-4533-be16-9d195a7f73c9\" (UID: \"47d2f169-c1d8-4533-be16-9d195a7f73c9\") " Oct 30 22:31:38.234038 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:38.233978 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47d2f169-c1d8-4533-be16-9d195a7f73c9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47d2f169-c1d8-4533-be16-9d195a7f73c9-istio-envoy\") pod \"47d2f169-c1d8-4533-be16-9d195a7f73c9\" (UID: \"47d2f169-c1d8-4533-be16-9d195a7f73c9\") " Oct 30 22:31:38.234038 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:38.233987 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47d2f169-c1d8-4533-be16-9d195a7f73c9-credential-socket" pod "47d2f169-c1d8-4533-be16-9d195a7f73c9" (UID: "47d2f169-c1d8-4533-be16-9d195a7f73c9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:38.234038 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:38.234020 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47d2f169-c1d8-4533-be16-9d195a7f73c9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47d2f169-c1d8-4533-be16-9d195a7f73c9-istiod-ca-cert\") pod \"47d2f169-c1d8-4533-be16-9d195a7f73c9\" (UID: \"47d2f169-c1d8-4533-be16-9d195a7f73c9\") " Oct 30 22:31:38.234565 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:38.234191 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47d2f169-c1d8-4533-be16-9d195a7f73c9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47d2f169-c1d8-4533-be16-9d195a7f73c9-workload-certs\") pod \"47d2f169-c1d8-4533-be16-9d195a7f73c9\" (UID: \"47d2f169-c1d8-4533-be16-9d195a7f73c9\") " Oct 30 22:31:38.234565 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:38.234197 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47d2f169-c1d8-4533-be16-9d195a7f73c9-workload-socket" pod "47d2f169-c1d8-4533-be16-9d195a7f73c9" (UID: "47d2f169-c1d8-4533-be16-9d195a7f73c9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:38.234878 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:38.234852 6720 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d2f169-c1d8-4533-be16-9d195a7f73c9-credential-socket\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:38.235074 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:38.235024 6720 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47d2f169-c1d8-4533-be16-9d195a7f73c9-workload-socket\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:38.235074 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:38.234865 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47d2f169-c1d8-4533-be16-9d195a7f73c9-workload-certs" pod "47d2f169-c1d8-4533-be16-9d195a7f73c9" (UID: "47d2f169-c1d8-4533-be16-9d195a7f73c9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:38.235222 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:38.235146 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47d2f169-c1d8-4533-be16-9d195a7f73c9-istiod-ca-cert" pod "47d2f169-c1d8-4533-be16-9d195a7f73c9" (UID: "47d2f169-c1d8-4533-be16-9d195a7f73c9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:31:38.235432 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:38.235408 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47d2f169-c1d8-4533-be16-9d195a7f73c9-istio-data" pod "47d2f169-c1d8-4533-be16-9d195a7f73c9" (UID: "47d2f169-c1d8-4533-be16-9d195a7f73c9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:38.237003 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:38.236971 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47d2f169-c1d8-4533-be16-9d195a7f73c9-kube-api-access-626xp" pod "47d2f169-c1d8-4533-be16-9d195a7f73c9" (UID: "47d2f169-c1d8-4533-be16-9d195a7f73c9"). InnerVolumeSpecName "kube-api-access-626xp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:38.237107 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:38.237063 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47d2f169-c1d8-4533-be16-9d195a7f73c9-istio-envoy" pod "47d2f169-c1d8-4533-be16-9d195a7f73c9" (UID: "47d2f169-c1d8-4533-be16-9d195a7f73c9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:38.237107 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:38.237083 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/47d2f169-c1d8-4533-be16-9d195a7f73c9-istio-podinfo" pod "47d2f169-c1d8-4533-be16-9d195a7f73c9" (UID: "47d2f169-c1d8-4533-be16-9d195a7f73c9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:31:38.237667 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:38.237648 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47d2f169-c1d8-4533-be16-9d195a7f73c9-istio-token" pod "47d2f169-c1d8-4533-be16-9d195a7f73c9" (UID: "47d2f169-c1d8-4533-be16-9d195a7f73c9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:38.336588 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:38.336550 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-626xp\" (UniqueName: \"kubernetes.io/projected/47d2f169-c1d8-4533-be16-9d195a7f73c9-kube-api-access-626xp\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:38.336588 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:38.336586 6720 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47d2f169-c1d8-4533-be16-9d195a7f73c9-istio-podinfo\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:38.336839 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:38.336601 6720 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47d2f169-c1d8-4533-be16-9d195a7f73c9-istio-token\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:38.336839 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:38.336616 6720 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47d2f169-c1d8-4533-be16-9d195a7f73c9-istio-data\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:38.336839 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:38.336630 6720 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47d2f169-c1d8-4533-be16-9d195a7f73c9-istio-envoy\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:38.336839 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:38.336646 6720 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47d2f169-c1d8-4533-be16-9d195a7f73c9-istiod-ca-cert\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:38.336839 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:38.336660 6720 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47d2f169-c1d8-4533-be16-9d195a7f73c9-workload-certs\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:38.423636 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:38.423364 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:38.509060 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:38.508981 6720 generic.go:356] "Generic (PLEG): container finished" podID="47d2f169-c1d8-4533-be16-9d195a7f73c9" containerID="713c79240f39a70ec0bbe2856a82666fd7cab78db9cfd6cbc110498580f9a2c8" exitCode=0 Oct 30 22:31:38.509060 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:38.509034 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-xbbpj" Oct 30 22:31:38.509246 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:38.509071 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-xbbpj" event={"ID":"47d2f169-c1d8-4533-be16-9d195a7f73c9","Type":"ContainerDied","Data":"713c79240f39a70ec0bbe2856a82666fd7cab78db9cfd6cbc110498580f9a2c8"} Oct 30 22:31:38.509246 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:38.509129 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-xbbpj" event={"ID":"47d2f169-c1d8-4533-be16-9d195a7f73c9","Type":"ContainerDied","Data":"f5e1b800305456071e237f3b3bcf27ad80aa68e9ca90ca6860c32932be4deef5"} Oct 30 22:31:38.509246 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:38.509159 6720 scope.go:122] "RemoveContainer" containerID="713c79240f39a70ec0bbe2856a82666fd7cab78db9cfd6cbc110498580f9a2c8" Oct 30 22:31:38.523379 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:38.523359 6720 scope.go:122] "RemoveContainer" containerID="713c79240f39a70ec0bbe2856a82666fd7cab78db9cfd6cbc110498580f9a2c8" Oct 30 22:31:38.523721 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:31:38.523698 6720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"713c79240f39a70ec0bbe2856a82666fd7cab78db9cfd6cbc110498580f9a2c8\": container with ID starting with 713c79240f39a70ec0bbe2856a82666fd7cab78db9cfd6cbc110498580f9a2c8 not found: ID does not exist" containerID="713c79240f39a70ec0bbe2856a82666fd7cab78db9cfd6cbc110498580f9a2c8" Oct 30 22:31:38.523793 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:38.523734 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"713c79240f39a70ec0bbe2856a82666fd7cab78db9cfd6cbc110498580f9a2c8"} err="failed to get container status \"713c79240f39a70ec0bbe2856a82666fd7cab78db9cfd6cbc110498580f9a2c8\": rpc error: code = NotFound desc = could not find container \"713c79240f39a70ec0bbe2856a82666fd7cab78db9cfd6cbc110498580f9a2c8\": container with ID starting with 713c79240f39a70ec0bbe2856a82666fd7cab78db9cfd6cbc110498580f9a2c8 not found: ID does not exist" Oct 30 22:31:38.539259 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:38.538700 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-xbbpj"] Oct 30 22:31:38.544041 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:38.544016 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-df200a-openshift-default-86d8f88f8f-xbbpj"] Oct 30 22:31:39.222149 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:39.218208 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:39.849327 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:39.844122 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:39.958599 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:39.958568 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47d2f169-c1d8-4533-be16-9d195a7f73c9" path="/var/lib/kubelet/pods/47d2f169-c1d8-4533-be16-9d195a7f73c9/volumes" Oct 30 22:31:40.404247 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:31:40.404206 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 22:31:40.536991 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:40.536910 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:41.208599 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:41.208563 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:41.542025 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:41.541920 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-4b2bb0/test-model-4b2bb0-kserve-5df44c7bc5-mj9p7" podUID="e7f48858-397b-46fe-84b4-f19ef4afa227" containerName="main" containerID="cri-o://8365e8b876cd67268b16869213046f1b7ad67025e0be94df18bd17822f578524" gracePeriod=45 Oct 30 22:31:41.834736 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:41.834701 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4b2bb0/test-model-4b2bb0-kserve-5df44c7bc5-mj9p7" Oct 30 22:31:41.883010 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:41.882971 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7f48858-397b-46fe-84b4-f19ef4afa227-kube-api-access-2xpx6\" (UniqueName: \"kubernetes.io/projected/e7f48858-397b-46fe-84b4-f19ef4afa227-kube-api-access-2xpx6\") pod \"e7f48858-397b-46fe-84b4-f19ef4afa227\" (UID: \"e7f48858-397b-46fe-84b4-f19ef4afa227\") " Oct 30 22:31:41.883197 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:41.883163 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7f48858-397b-46fe-84b4-f19ef4afa227-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7f48858-397b-46fe-84b4-f19ef4afa227-dshm\") pod \"e7f48858-397b-46fe-84b4-f19ef4afa227\" (UID: \"e7f48858-397b-46fe-84b4-f19ef4afa227\") " Oct 30 22:31:41.883260 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:41.883212 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7f48858-397b-46fe-84b4-f19ef4afa227-home\" (UniqueName: \"kubernetes.io/empty-dir/e7f48858-397b-46fe-84b4-f19ef4afa227-home\") pod \"e7f48858-397b-46fe-84b4-f19ef4afa227\" (UID: \"e7f48858-397b-46fe-84b4-f19ef4afa227\") " Oct 30 22:31:41.883260 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:41.883248 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7f48858-397b-46fe-84b4-f19ef4afa227-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7f48858-397b-46fe-84b4-f19ef4afa227-tmp-dir\") pod \"e7f48858-397b-46fe-84b4-f19ef4afa227\" (UID: \"e7f48858-397b-46fe-84b4-f19ef4afa227\") " Oct 30 22:31:41.883354 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:41.883323 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7f48858-397b-46fe-84b4-f19ef4afa227-tls-certs\" (UniqueName: \"kubernetes.io/secret/e7f48858-397b-46fe-84b4-f19ef4afa227-tls-certs\") pod \"e7f48858-397b-46fe-84b4-f19ef4afa227\" (UID: \"e7f48858-397b-46fe-84b4-f19ef4afa227\") " Oct 30 22:31:41.884289 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:41.883445 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7f48858-397b-46fe-84b4-f19ef4afa227-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7f48858-397b-46fe-84b4-f19ef4afa227-model-cache\") pod \"e7f48858-397b-46fe-84b4-f19ef4afa227\" (UID: \"e7f48858-397b-46fe-84b4-f19ef4afa227\") " Oct 30 22:31:41.884289 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:41.883627 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7f48858-397b-46fe-84b4-f19ef4afa227-home" pod "e7f48858-397b-46fe-84b4-f19ef4afa227" (UID: "e7f48858-397b-46fe-84b4-f19ef4afa227"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:41.884289 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:41.883962 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7f48858-397b-46fe-84b4-f19ef4afa227-tmp-dir" pod "e7f48858-397b-46fe-84b4-f19ef4afa227" (UID: "e7f48858-397b-46fe-84b4-f19ef4afa227"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:41.884289 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:41.884184 6720 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7f48858-397b-46fe-84b4-f19ef4afa227-home\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:41.884289 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:41.884207 6720 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7f48858-397b-46fe-84b4-f19ef4afa227-tmp-dir\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:41.884289 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:41.884236 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7f48858-397b-46fe-84b4-f19ef4afa227-model-cache" pod "e7f48858-397b-46fe-84b4-f19ef4afa227" (UID: "e7f48858-397b-46fe-84b4-f19ef4afa227"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:41.887653 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:41.887618 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7f48858-397b-46fe-84b4-f19ef4afa227-kube-api-access-2xpx6" pod "e7f48858-397b-46fe-84b4-f19ef4afa227" (UID: "e7f48858-397b-46fe-84b4-f19ef4afa227"). InnerVolumeSpecName "kube-api-access-2xpx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:41.887864 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:41.887837 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7f48858-397b-46fe-84b4-f19ef4afa227-tls-certs" pod "e7f48858-397b-46fe-84b4-f19ef4afa227" (UID: "e7f48858-397b-46fe-84b4-f19ef4afa227"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:31:41.888227 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:41.888196 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7f48858-397b-46fe-84b4-f19ef4afa227-dshm" pod "e7f48858-397b-46fe-84b4-f19ef4afa227" (UID: "e7f48858-397b-46fe-84b4-f19ef4afa227"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:41.956879 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:41.956835 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:41.986423 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:41.986342 6720 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7f48858-397b-46fe-84b4-f19ef4afa227-model-cache\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:41.986423 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:41.986379 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xpx6\" (UniqueName: \"kubernetes.io/projected/e7f48858-397b-46fe-84b4-f19ef4afa227-kube-api-access-2xpx6\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:41.986423 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:41.986415 6720 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7f48858-397b-46fe-84b4-f19ef4afa227-dshm\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:41.986423 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:41.986429 6720 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7f48858-397b-46fe-84b4-f19ef4afa227-tls-certs\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:42.130707 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:42.128622 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:42.287653 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:42.285687 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:42.556197 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:42.556099 6720 generic.go:356] "Generic (PLEG): container finished" podID="e7f48858-397b-46fe-84b4-f19ef4afa227" containerID="8365e8b876cd67268b16869213046f1b7ad67025e0be94df18bd17822f578524" exitCode=0 Oct 30 22:31:42.556197 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:42.556141 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4b2bb0/test-model-4b2bb0-kserve-5df44c7bc5-mj9p7" Oct 30 22:31:42.556197 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:42.556169 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4b2bb0/test-model-4b2bb0-kserve-5df44c7bc5-mj9p7" event={"ID":"e7f48858-397b-46fe-84b4-f19ef4afa227","Type":"ContainerDied","Data":"8365e8b876cd67268b16869213046f1b7ad67025e0be94df18bd17822f578524"} Oct 30 22:31:42.556885 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:42.556232 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4b2bb0/test-model-4b2bb0-kserve-5df44c7bc5-mj9p7" event={"ID":"e7f48858-397b-46fe-84b4-f19ef4afa227","Type":"ContainerDied","Data":"61656d44f52ced7df7983a05a833ff51f651ad5b32d9e5960ad7689acd8ed94d"} Oct 30 22:31:42.556885 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:42.556258 6720 scope.go:122] "RemoveContainer" containerID="8365e8b876cd67268b16869213046f1b7ad67025e0be94df18bd17822f578524" Oct 30 22:31:42.568162 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:42.568144 6720 scope.go:122] "RemoveContainer" containerID="8365e8b876cd67268b16869213046f1b7ad67025e0be94df18bd17822f578524" Oct 30 22:31:42.568453 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:31:42.568434 6720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8365e8b876cd67268b16869213046f1b7ad67025e0be94df18bd17822f578524\": container with ID starting with 8365e8b876cd67268b16869213046f1b7ad67025e0be94df18bd17822f578524 not found: ID does not exist" containerID="8365e8b876cd67268b16869213046f1b7ad67025e0be94df18bd17822f578524" Oct 30 22:31:42.568513 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:42.568462 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8365e8b876cd67268b16869213046f1b7ad67025e0be94df18bd17822f578524"} err="failed to get container status \"8365e8b876cd67268b16869213046f1b7ad67025e0be94df18bd17822f578524\": rpc error: code = NotFound desc = could not find container \"8365e8b876cd67268b16869213046f1b7ad67025e0be94df18bd17822f578524\": container with ID starting with 8365e8b876cd67268b16869213046f1b7ad67025e0be94df18bd17822f578524 not found: ID does not exist" Oct 30 22:31:42.583211 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:42.583187 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-4b2bb0/test-model-4b2bb0-kserve-5df44c7bc5-mj9p7"] Oct 30 22:31:42.589573 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:42.589551 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-4b2bb0/test-model-4b2bb0-kserve-5df44c7bc5-mj9p7"] Oct 30 22:31:42.683412 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:42.678618 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:43.152749 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:43.152720 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-120cb1-6897f8fbbb-jszpt"] Oct 30 22:31:43.152995 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:43.152970 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-120cb1-6897f8fbbb-jszpt" podUID="24a24a49-0696-4e41-8b93-0b11814f98cd" containerName="maas-api" containerID="cri-o://591a61d66cf60d06f073464299527bc4fab4938da2a831c4c2198c4417400fc1" gracePeriod=30 Oct 30 22:31:43.224233 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:43.224194 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:43.381173 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:43.381065 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:43.553425 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:43.546808 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:43.601824 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:43.601426 6720 generic.go:356] "Generic (PLEG): container finished" podID="24a24a49-0696-4e41-8b93-0b11814f98cd" containerID="591a61d66cf60d06f073464299527bc4fab4938da2a831c4c2198c4417400fc1" exitCode=0 Oct 30 22:31:43.601824 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:43.601610 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-120cb1-6897f8fbbb-jszpt" event={"ID":"24a24a49-0696-4e41-8b93-0b11814f98cd","Type":"ContainerDied","Data":"591a61d66cf60d06f073464299527bc4fab4938da2a831c4c2198c4417400fc1"} Oct 30 22:31:43.663418 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:43.663361 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-120cb1-6897f8fbbb-jszpt" Oct 30 22:31:43.758614 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:43.758532 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:43.819892 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:43.819857 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24a24a49-0696-4e41-8b93-0b11814f98cd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/24a24a49-0696-4e41-8b93-0b11814f98cd-metrics-certs\") pod \"24a24a49-0696-4e41-8b93-0b11814f98cd\" (UID: \"24a24a49-0696-4e41-8b93-0b11814f98cd\") " Oct 30 22:31:43.820157 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:43.820134 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24a24a49-0696-4e41-8b93-0b11814f98cd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24a24a49-0696-4e41-8b93-0b11814f98cd-maas-api-tls\") pod \"24a24a49-0696-4e41-8b93-0b11814f98cd\" (UID: \"24a24a49-0696-4e41-8b93-0b11814f98cd\") " Oct 30 22:31:43.820259 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:43.820195 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24a24a49-0696-4e41-8b93-0b11814f98cd-kube-api-access-2l47m\" (UniqueName: \"kubernetes.io/projected/24a24a49-0696-4e41-8b93-0b11814f98cd-kube-api-access-2l47m\") pod \"24a24a49-0696-4e41-8b93-0b11814f98cd\" (UID: \"24a24a49-0696-4e41-8b93-0b11814f98cd\") " Oct 30 22:31:43.820349 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:43.820325 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24a24a49-0696-4e41-8b93-0b11814f98cd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24a24a49-0696-4e41-8b93-0b11814f98cd-openshift-service-ca\") pod \"24a24a49-0696-4e41-8b93-0b11814f98cd\" (UID: \"24a24a49-0696-4e41-8b93-0b11814f98cd\") " Oct 30 22:31:43.821378 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:43.821344 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24a24a49-0696-4e41-8b93-0b11814f98cd-openshift-service-ca" pod "24a24a49-0696-4e41-8b93-0b11814f98cd" (UID: "24a24a49-0696-4e41-8b93-0b11814f98cd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:31:43.843463 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:43.838924 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24a24a49-0696-4e41-8b93-0b11814f98cd-metrics-certs" pod "24a24a49-0696-4e41-8b93-0b11814f98cd" (UID: "24a24a49-0696-4e41-8b93-0b11814f98cd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:31:43.843463 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:43.839780 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24a24a49-0696-4e41-8b93-0b11814f98cd-maas-api-tls" pod "24a24a49-0696-4e41-8b93-0b11814f98cd" (UID: "24a24a49-0696-4e41-8b93-0b11814f98cd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:31:43.843463 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:43.839872 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24a24a49-0696-4e41-8b93-0b11814f98cd-kube-api-access-2l47m" pod "24a24a49-0696-4e41-8b93-0b11814f98cd" (UID: "24a24a49-0696-4e41-8b93-0b11814f98cd"). InnerVolumeSpecName "kube-api-access-2l47m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:43.909887 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:43.909855 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:43.922234 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:43.922207 6720 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24a24a49-0696-4e41-8b93-0b11814f98cd-metrics-certs\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:43.922234 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:43.922231 6720 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24a24a49-0696-4e41-8b93-0b11814f98cd-maas-api-tls\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:43.922417 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:43.922242 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2l47m\" (UniqueName: \"kubernetes.io/projected/24a24a49-0696-4e41-8b93-0b11814f98cd-kube-api-access-2l47m\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:43.922417 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:43.922252 6720 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24a24a49-0696-4e41-8b93-0b11814f98cd-openshift-service-ca\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:43.958617 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:43.958584 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7f48858-397b-46fe-84b4-f19ef4afa227" path="/var/lib/kubelet/pods/e7f48858-397b-46fe-84b4-f19ef4afa227/volumes" Oct 30 22:31:44.065500 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:44.060470 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:44.218926 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:44.218890 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:44.550476 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:44.550433 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:44.609481 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:44.609437 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-120cb1-6897f8fbbb-jszpt" event={"ID":"24a24a49-0696-4e41-8b93-0b11814f98cd","Type":"ContainerDied","Data":"d3e27568b8a3633ce0f2b30ce325a34381d0f49a52093c43d9266d3b4b67cbc2"} Oct 30 22:31:44.609955 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:44.609511 6720 scope.go:122] "RemoveContainer" containerID="591a61d66cf60d06f073464299527bc4fab4938da2a831c4c2198c4417400fc1" Oct 30 22:31:44.609955 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:44.609454 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-120cb1-6897f8fbbb-jszpt" Oct 30 22:31:44.629432 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:44.629405 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-120cb1-6897f8fbbb-jszpt"] Oct 30 22:31:44.633921 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:44.633897 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-120cb1-6897f8fbbb-jszpt"] Oct 30 22:31:45.447488 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:45.445972 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:45.958751 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:45.958706 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24a24a49-0696-4e41-8b93-0b11814f98cd" path="/var/lib/kubelet/pods/24a24a49-0696-4e41-8b93-0b11814f98cd/volumes" Oct 30 22:31:46.126456 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:46.126424 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:46.893977 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:46.893941 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:47.360812 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:47.360781 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-58d5fa-58b5b7b7df-nzxwm"] Oct 30 22:31:47.361318 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:47.361049 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58d5fa-58b5b7b7df-nzxwm" podUID="b6c2aa19-cad5-435a-a573-56146651be06" containerName="maas-api" containerID="cri-o://e974e2b2a84cb8b12c42b612230e51f70f231ae888cb9efad5226d1bcffe7769" gracePeriod=30 Oct 30 22:31:47.453653 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:47.453617 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:47.568273 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:47.567932 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58d5fa-58b5b7b7df-nzxwm" podUID="b6c2aa19-cad5-435a-a573-56146651be06" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.128:8443/health\": dial tcp 10.134.0.128:8443: connect: connection refused" Oct 30 22:31:47.618461 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:47.611909 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:47.809502 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:47.809465 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:47.845808 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:47.840745 6720 generic.go:356] "Generic (PLEG): container finished" podID="b6c2aa19-cad5-435a-a573-56146651be06" containerID="e974e2b2a84cb8b12c42b612230e51f70f231ae888cb9efad5226d1bcffe7769" exitCode=0 Oct 30 22:31:47.845808 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:47.840904 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58d5fa-58b5b7b7df-nzxwm" event={"ID":"b6c2aa19-cad5-435a-a573-56146651be06","Type":"ContainerDied","Data":"e974e2b2a84cb8b12c42b612230e51f70f231ae888cb9efad5226d1bcffe7769"} Oct 30 22:31:47.904228 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:47.904203 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58d5fa-58b5b7b7df-nzxwm" Oct 30 22:31:47.980232 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:47.979972 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:48.023188 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:48.023158 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6c2aa19-cad5-435a-a573-56146651be06-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6c2aa19-cad5-435a-a573-56146651be06-maas-api-tls\") pod \"b6c2aa19-cad5-435a-a573-56146651be06\" (UID: \"b6c2aa19-cad5-435a-a573-56146651be06\") " Oct 30 22:31:48.023372 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:48.023230 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6c2aa19-cad5-435a-a573-56146651be06-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6c2aa19-cad5-435a-a573-56146651be06-openshift-service-ca\") pod \"b6c2aa19-cad5-435a-a573-56146651be06\" (UID: \"b6c2aa19-cad5-435a-a573-56146651be06\") " Oct 30 22:31:48.023372 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:48.023341 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6c2aa19-cad5-435a-a573-56146651be06-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6c2aa19-cad5-435a-a573-56146651be06-metrics-certs\") pod \"b6c2aa19-cad5-435a-a573-56146651be06\" (UID: \"b6c2aa19-cad5-435a-a573-56146651be06\") " Oct 30 22:31:48.023372 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:48.023363 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6c2aa19-cad5-435a-a573-56146651be06-kube-api-access-n475c\" (UniqueName: \"kubernetes.io/projected/b6c2aa19-cad5-435a-a573-56146651be06-kube-api-access-n475c\") pod \"b6c2aa19-cad5-435a-a573-56146651be06\" (UID: \"b6c2aa19-cad5-435a-a573-56146651be06\") " Oct 30 22:31:48.023654 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:48.023629 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6c2aa19-cad5-435a-a573-56146651be06-openshift-service-ca" pod "b6c2aa19-cad5-435a-a573-56146651be06" (UID: "b6c2aa19-cad5-435a-a573-56146651be06"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:31:48.023945 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:48.023924 6720 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6c2aa19-cad5-435a-a573-56146651be06-openshift-service-ca\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:48.025524 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:48.025498 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6c2aa19-cad5-435a-a573-56146651be06-metrics-certs" pod "b6c2aa19-cad5-435a-a573-56146651be06" (UID: "b6c2aa19-cad5-435a-a573-56146651be06"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:31:48.025602 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:48.025548 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6c2aa19-cad5-435a-a573-56146651be06-maas-api-tls" pod "b6c2aa19-cad5-435a-a573-56146651be06" (UID: "b6c2aa19-cad5-435a-a573-56146651be06"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:31:48.025650 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:48.025605 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6c2aa19-cad5-435a-a573-56146651be06-kube-api-access-n475c" pod "b6c2aa19-cad5-435a-a573-56146651be06" (UID: "b6c2aa19-cad5-435a-a573-56146651be06"). InnerVolumeSpecName "kube-api-access-n475c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:48.125174 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:48.125136 6720 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6c2aa19-cad5-435a-a573-56146651be06-maas-api-tls\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:48.125174 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:48.125164 6720 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6c2aa19-cad5-435a-a573-56146651be06-metrics-certs\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:48.125174 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:48.125179 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n475c\" (UniqueName: \"kubernetes.io/projected/b6c2aa19-cad5-435a-a573-56146651be06-kube-api-access-n475c\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:48.281112 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:48.281028 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:48.723314 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:48.723279 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:48.848173 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:48.848144 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58d5fa-58b5b7b7df-nzxwm" Oct 30 22:31:48.848173 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:48.848160 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-58d5fa-58b5b7b7df-nzxwm" event={"ID":"b6c2aa19-cad5-435a-a573-56146651be06","Type":"ContainerDied","Data":"13ab8eca3f040397cf61ddf50fa6759540691faadd4fdddf78f37afac134649a"} Oct 30 22:31:48.848369 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:48.848217 6720 scope.go:122] "RemoveContainer" containerID="e974e2b2a84cb8b12c42b612230e51f70f231ae888cb9efad5226d1bcffe7769" Oct 30 22:31:48.873535 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:48.873508 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-58d5fa-58b5b7b7df-nzxwm"] Oct 30 22:31:48.877753 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:48.877726 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-58d5fa-58b5b7b7df-nzxwm"] Oct 30 22:31:49.363794 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:49.363761 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:49.957580 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:49.957548 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6c2aa19-cad5-435a-a573-56146651be06" path="/var/lib/kubelet/pods/b6c2aa19-cad5-435a-a573-56146651be06/volumes" Oct 30 22:31:50.145909 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:50.145589 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:50.868544 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:50.868488 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:51.022140 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:51.022061 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:51.153187 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:51.153093 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-vnctc"] Oct 30 22:31:51.153365 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:51.153190 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-v4kbr"] Oct 30 22:31:51.153483 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:51.153452 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-v4kbr" podUID="744e223f-d9d1-45c3-9f37-b4d29091852c" containerName="istio-proxy" containerID="cri-o://bedcecf53043ca5387632a0ffd82fb1bb5179d55f20c8e358e8bf667d5e3237d" gracePeriod=30 Oct 30 22:31:51.153924 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:51.153887 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-vnctc" podUID="961c219a-a80f-46eb-ab6c-2920a28419e2" containerName="istio-proxy" containerID="cri-o://093ba049dfdd3a8f1194b04b2e1b09bc49974a3d64ca79369d6482d8f22859ce" gracePeriod=30 Oct 30 22:31:51.179905 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:51.179881 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-x64z9"] Oct 30 22:31:51.180071 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:51.180053 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-x64z9" podUID="ea5efed8-2dea-46bf-9162-55fe6a1459d5" containerName="istio-proxy" containerID="cri-o://186fdb74a0368dc7020f7e17c2afefa3e66b060f9f47ec8464d20ee09504852c" gracePeriod=30 Oct 30 22:31:51.347330 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:51.347288 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:51.668262 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:51.668205 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c92aec-5bd6c598b7-f86vh"] Oct 30 22:31:51.668482 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:51.668416 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c92aec-5bd6c598b7-f86vh" podUID="22545d04-38b5-4d64-89a0-9e533875220e" containerName="maas-api" containerID="cri-o://4aa532de7b40bd6d0e1e14da80f5f23efed42aa70402a94491e4bc801be879b0" gracePeriod=30 Oct 30 22:31:51.782312 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:51.782276 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:51.894102 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:51.892302 6720 generic.go:356] "Generic (PLEG): container finished" podID="22545d04-38b5-4d64-89a0-9e533875220e" containerID="4aa532de7b40bd6d0e1e14da80f5f23efed42aa70402a94491e4bc801be879b0" exitCode=0 Oct 30 22:31:51.894102 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:51.892348 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c92aec-5bd6c598b7-f86vh" event={"ID":"22545d04-38b5-4d64-89a0-9e533875220e","Type":"ContainerDied","Data":"4aa532de7b40bd6d0e1e14da80f5f23efed42aa70402a94491e4bc801be879b0"} Oct 30 22:31:52.076671 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:52.048700 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:52.132514 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:52.132487 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c92aec-5bd6c598b7-f86vh" Oct 30 22:31:52.290093 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:52.290026 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:52.305228 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:52.302717 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22545d04-38b5-4d64-89a0-9e533875220e-kube-api-access-62pgj\" (UniqueName: \"kubernetes.io/projected/22545d04-38b5-4d64-89a0-9e533875220e-kube-api-access-62pgj\") pod \"22545d04-38b5-4d64-89a0-9e533875220e\" (UID: \"22545d04-38b5-4d64-89a0-9e533875220e\") " Oct 30 22:31:52.305228 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:52.302817 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22545d04-38b5-4d64-89a0-9e533875220e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22545d04-38b5-4d64-89a0-9e533875220e-maas-api-tls\") pod \"22545d04-38b5-4d64-89a0-9e533875220e\" (UID: \"22545d04-38b5-4d64-89a0-9e533875220e\") " Oct 30 22:31:52.305228 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:52.302878 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22545d04-38b5-4d64-89a0-9e533875220e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/22545d04-38b5-4d64-89a0-9e533875220e-metrics-certs\") pod \"22545d04-38b5-4d64-89a0-9e533875220e\" (UID: \"22545d04-38b5-4d64-89a0-9e533875220e\") " Oct 30 22:31:52.305228 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:52.302968 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22545d04-38b5-4d64-89a0-9e533875220e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22545d04-38b5-4d64-89a0-9e533875220e-openshift-service-ca\") pod \"22545d04-38b5-4d64-89a0-9e533875220e\" (UID: \"22545d04-38b5-4d64-89a0-9e533875220e\") " Oct 30 22:31:52.305228 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:52.303468 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22545d04-38b5-4d64-89a0-9e533875220e-openshift-service-ca" pod "22545d04-38b5-4d64-89a0-9e533875220e" (UID: "22545d04-38b5-4d64-89a0-9e533875220e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:31:52.305228 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:52.303644 6720 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22545d04-38b5-4d64-89a0-9e533875220e-openshift-service-ca\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:52.305779 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:52.305754 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22545d04-38b5-4d64-89a0-9e533875220e-metrics-certs" pod "22545d04-38b5-4d64-89a0-9e533875220e" (UID: "22545d04-38b5-4d64-89a0-9e533875220e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:31:52.305844 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:52.305780 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22545d04-38b5-4d64-89a0-9e533875220e-kube-api-access-62pgj" pod "22545d04-38b5-4d64-89a0-9e533875220e" (UID: "22545d04-38b5-4d64-89a0-9e533875220e"). InnerVolumeSpecName "kube-api-access-62pgj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:52.306808 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:52.306783 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22545d04-38b5-4d64-89a0-9e533875220e-maas-api-tls" pod "22545d04-38b5-4d64-89a0-9e533875220e" (UID: "22545d04-38b5-4d64-89a0-9e533875220e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:31:52.408776 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:52.405007 6720 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22545d04-38b5-4d64-89a0-9e533875220e-maas-api-tls\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:52.408776 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:52.405041 6720 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22545d04-38b5-4d64-89a0-9e533875220e-metrics-certs\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:52.408776 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:52.405058 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62pgj\" (UniqueName: \"kubernetes.io/projected/22545d04-38b5-4d64-89a0-9e533875220e-kube-api-access-62pgj\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:52.457200 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:52.457165 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:52.906027 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:52.905986 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c92aec-5bd6c598b7-f86vh" Oct 30 22:31:52.906249 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:52.906039 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c92aec-5bd6c598b7-f86vh" event={"ID":"22545d04-38b5-4d64-89a0-9e533875220e","Type":"ContainerDied","Data":"9c072aba05841381dee38d644409505ce32dbf4e3ae16d0ac82f13d46ca9ae30"} Oct 30 22:31:52.906249 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:52.906106 6720 scope.go:122] "RemoveContainer" containerID="4aa532de7b40bd6d0e1e14da80f5f23efed42aa70402a94491e4bc801be879b0" Oct 30 22:31:52.918525 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:52.918500 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:52.939714 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:52.939692 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c92aec-5bd6c598b7-f86vh"] Oct 30 22:31:52.947280 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:52.947251 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c92aec-5bd6c598b7-f86vh"] Oct 30 22:31:53.427409 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:53.422713 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:53.957925 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:53.957896 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22545d04-38b5-4d64-89a0-9e533875220e" path="/var/lib/kubelet/pods/22545d04-38b5-4d64-89a0-9e533875220e/volumes" Oct 30 22:31:54.054499 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:54.054462 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:54.748546 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:54.746419 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:55.431920 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:55.431869 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:55.602355 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:55.602273 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:55.927537 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:55.927503 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:56.242974 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:31:56.242943 6720 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod744e223f_d9d1_45c3_9f37_b4d29091852c.slice/crio-bedcecf53043ca5387632a0ffd82fb1bb5179d55f20c8e358e8bf667d5e3237d.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea5efed8_2dea_46bf_9162_55fe6a1459d5.slice/crio-186fdb74a0368dc7020f7e17c2afefa3e66b060f9f47ec8464d20ee09504852c.scope\": RecentStats: unable to find data in memory cache]" Oct 30 22:31:56.426901 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.426862 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:56.527071 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.527047 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-v4kbr" Oct 30 22:31:56.552164 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.552140 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-x64z9" Oct 30 22:31:56.597492 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.597466 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-vnctc" Oct 30 22:31:56.618259 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.618229 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/744e223f-d9d1-45c3-9f37-b4d29091852c-istio-token\" (UniqueName: \"kubernetes.io/projected/744e223f-d9d1-45c3-9f37-b4d29091852c-istio-token\") pod \"744e223f-d9d1-45c3-9f37-b4d29091852c\" (UID: \"744e223f-d9d1-45c3-9f37-b4d29091852c\") " Oct 30 22:31:56.618504 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.618475 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea5efed8-2dea-46bf-9162-55fe6a1459d5-kube-api-access-kj84j\" (UniqueName: \"kubernetes.io/projected/ea5efed8-2dea-46bf-9162-55fe6a1459d5-kube-api-access-kj84j\") pod \"ea5efed8-2dea-46bf-9162-55fe6a1459d5\" (UID: \"ea5efed8-2dea-46bf-9162-55fe6a1459d5\") " Oct 30 22:31:56.618633 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.618559 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/744e223f-d9d1-45c3-9f37-b4d29091852c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/744e223f-d9d1-45c3-9f37-b4d29091852c-workload-socket\") pod \"744e223f-d9d1-45c3-9f37-b4d29091852c\" (UID: \"744e223f-d9d1-45c3-9f37-b4d29091852c\") " Oct 30 22:31:56.618633 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.618595 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ea5efed8-2dea-46bf-9162-55fe6a1459d5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea5efed8-2dea-46bf-9162-55fe6a1459d5-istio-podinfo\") pod \"ea5efed8-2dea-46bf-9162-55fe6a1459d5\" (UID: \"ea5efed8-2dea-46bf-9162-55fe6a1459d5\") " Oct 30 22:31:56.618774 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.618640 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/744e223f-d9d1-45c3-9f37-b4d29091852c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/744e223f-d9d1-45c3-9f37-b4d29091852c-istiod-ca-cert\") pod \"744e223f-d9d1-45c3-9f37-b4d29091852c\" (UID: \"744e223f-d9d1-45c3-9f37-b4d29091852c\") " Oct 30 22:31:56.618774 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.618692 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea5efed8-2dea-46bf-9162-55fe6a1459d5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea5efed8-2dea-46bf-9162-55fe6a1459d5-workload-socket\") pod \"ea5efed8-2dea-46bf-9162-55fe6a1459d5\" (UID: \"ea5efed8-2dea-46bf-9162-55fe6a1459d5\") " Oct 30 22:31:56.618774 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.618738 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/744e223f-d9d1-45c3-9f37-b4d29091852c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/744e223f-d9d1-45c3-9f37-b4d29091852c-istio-data\") pod \"744e223f-d9d1-45c3-9f37-b4d29091852c\" (UID: \"744e223f-d9d1-45c3-9f37-b4d29091852c\") " Oct 30 22:31:56.618946 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.618781 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea5efed8-2dea-46bf-9162-55fe6a1459d5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea5efed8-2dea-46bf-9162-55fe6a1459d5-credential-socket\") pod \"ea5efed8-2dea-46bf-9162-55fe6a1459d5\" (UID: \"ea5efed8-2dea-46bf-9162-55fe6a1459d5\") " Oct 30 22:31:56.618946 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.618826 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/744e223f-d9d1-45c3-9f37-b4d29091852c-kube-api-access-wnk7n\" (UniqueName: \"kubernetes.io/projected/744e223f-d9d1-45c3-9f37-b4d29091852c-kube-api-access-wnk7n\") pod \"744e223f-d9d1-45c3-9f37-b4d29091852c\" (UID: \"744e223f-d9d1-45c3-9f37-b4d29091852c\") " Oct 30 22:31:56.618946 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.618883 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/744e223f-d9d1-45c3-9f37-b4d29091852c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/744e223f-d9d1-45c3-9f37-b4d29091852c-istio-envoy\") pod \"744e223f-d9d1-45c3-9f37-b4d29091852c\" (UID: \"744e223f-d9d1-45c3-9f37-b4d29091852c\") " Oct 30 22:31:56.619113 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.618969 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea5efed8-2dea-46bf-9162-55fe6a1459d5-istio-token\" (UniqueName: \"kubernetes.io/projected/ea5efed8-2dea-46bf-9162-55fe6a1459d5-istio-token\") pod \"ea5efed8-2dea-46bf-9162-55fe6a1459d5\" (UID: \"ea5efed8-2dea-46bf-9162-55fe6a1459d5\") " Oct 30 22:31:56.619113 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.618998 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/744e223f-d9d1-45c3-9f37-b4d29091852c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/744e223f-d9d1-45c3-9f37-b4d29091852c-istio-podinfo\") pod \"744e223f-d9d1-45c3-9f37-b4d29091852c\" (UID: \"744e223f-d9d1-45c3-9f37-b4d29091852c\") " Oct 30 22:31:56.619113 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.619082 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea5efed8-2dea-46bf-9162-55fe6a1459d5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea5efed8-2dea-46bf-9162-55fe6a1459d5-istio-envoy\") pod \"ea5efed8-2dea-46bf-9162-55fe6a1459d5\" (UID: \"ea5efed8-2dea-46bf-9162-55fe6a1459d5\") " Oct 30 22:31:56.619283 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.619117 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea5efed8-2dea-46bf-9162-55fe6a1459d5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea5efed8-2dea-46bf-9162-55fe6a1459d5-istio-data\") pod \"ea5efed8-2dea-46bf-9162-55fe6a1459d5\" (UID: \"ea5efed8-2dea-46bf-9162-55fe6a1459d5\") " Oct 30 22:31:56.619283 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.619149 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea5efed8-2dea-46bf-9162-55fe6a1459d5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea5efed8-2dea-46bf-9162-55fe6a1459d5-workload-certs\") pod \"ea5efed8-2dea-46bf-9162-55fe6a1459d5\" (UID: \"ea5efed8-2dea-46bf-9162-55fe6a1459d5\") " Oct 30 22:31:56.619283 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.619175 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/744e223f-d9d1-45c3-9f37-b4d29091852c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/744e223f-d9d1-45c3-9f37-b4d29091852c-workload-certs\") pod \"744e223f-d9d1-45c3-9f37-b4d29091852c\" (UID: \"744e223f-d9d1-45c3-9f37-b4d29091852c\") " Oct 30 22:31:56.619283 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.619199 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea5efed8-2dea-46bf-9162-55fe6a1459d5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea5efed8-2dea-46bf-9162-55fe6a1459d5-istiod-ca-cert\") pod \"ea5efed8-2dea-46bf-9162-55fe6a1459d5\" (UID: \"ea5efed8-2dea-46bf-9162-55fe6a1459d5\") " Oct 30 22:31:56.619283 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.619232 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/744e223f-d9d1-45c3-9f37-b4d29091852c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/744e223f-d9d1-45c3-9f37-b4d29091852c-credential-socket\") pod \"744e223f-d9d1-45c3-9f37-b4d29091852c\" (UID: \"744e223f-d9d1-45c3-9f37-b4d29091852c\") " Oct 30 22:31:56.620062 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.620040 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/744e223f-d9d1-45c3-9f37-b4d29091852c-credential-socket" pod "744e223f-d9d1-45c3-9f37-b4d29091852c" (UID: "744e223f-d9d1-45c3-9f37-b4d29091852c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:56.621603 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.621573 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea5efed8-2dea-46bf-9162-55fe6a1459d5-istio-data" pod "ea5efed8-2dea-46bf-9162-55fe6a1459d5" (UID: "ea5efed8-2dea-46bf-9162-55fe6a1459d5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:56.621943 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.621918 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea5efed8-2dea-46bf-9162-55fe6a1459d5-credential-socket" pod "ea5efed8-2dea-46bf-9162-55fe6a1459d5" (UID: "ea5efed8-2dea-46bf-9162-55fe6a1459d5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:56.622144 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.622121 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/744e223f-d9d1-45c3-9f37-b4d29091852c-istio-envoy" pod "744e223f-d9d1-45c3-9f37-b4d29091852c" (UID: "744e223f-d9d1-45c3-9f37-b4d29091852c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:56.622260 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.622206 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/744e223f-d9d1-45c3-9f37-b4d29091852c-istio-data" pod "744e223f-d9d1-45c3-9f37-b4d29091852c" (UID: "744e223f-d9d1-45c3-9f37-b4d29091852c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:56.622574 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.622551 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea5efed8-2dea-46bf-9162-55fe6a1459d5-istio-token" pod "ea5efed8-2dea-46bf-9162-55fe6a1459d5" (UID: "ea5efed8-2dea-46bf-9162-55fe6a1459d5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:56.622704 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.622513 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea5efed8-2dea-46bf-9162-55fe6a1459d5-workload-socket" pod "ea5efed8-2dea-46bf-9162-55fe6a1459d5" (UID: "ea5efed8-2dea-46bf-9162-55fe6a1459d5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:56.622807 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.622783 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea5efed8-2dea-46bf-9162-55fe6a1459d5-workload-certs" pod "ea5efed8-2dea-46bf-9162-55fe6a1459d5" (UID: "ea5efed8-2dea-46bf-9162-55fe6a1459d5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:56.622873 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.622593 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/744e223f-d9d1-45c3-9f37-b4d29091852c-workload-certs" pod "744e223f-d9d1-45c3-9f37-b4d29091852c" (UID: "744e223f-d9d1-45c3-9f37-b4d29091852c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:56.622873 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.622846 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/744e223f-d9d1-45c3-9f37-b4d29091852c-istiod-ca-cert" pod "744e223f-d9d1-45c3-9f37-b4d29091852c" (UID: "744e223f-d9d1-45c3-9f37-b4d29091852c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:31:56.623014 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.622993 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/744e223f-d9d1-45c3-9f37-b4d29091852c-workload-socket" pod "744e223f-d9d1-45c3-9f37-b4d29091852c" (UID: "744e223f-d9d1-45c3-9f37-b4d29091852c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:56.623360 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.623339 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea5efed8-2dea-46bf-9162-55fe6a1459d5-istiod-ca-cert" pod "ea5efed8-2dea-46bf-9162-55fe6a1459d5" (UID: "ea5efed8-2dea-46bf-9162-55fe6a1459d5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:31:56.624365 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.624332 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/744e223f-d9d1-45c3-9f37-b4d29091852c-kube-api-access-wnk7n" pod "744e223f-d9d1-45c3-9f37-b4d29091852c" (UID: "744e223f-d9d1-45c3-9f37-b4d29091852c"). InnerVolumeSpecName "kube-api-access-wnk7n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:56.625992 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.625602 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ea5efed8-2dea-46bf-9162-55fe6a1459d5-istio-podinfo" pod "ea5efed8-2dea-46bf-9162-55fe6a1459d5" (UID: "ea5efed8-2dea-46bf-9162-55fe6a1459d5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:31:56.625992 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.625698 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/744e223f-d9d1-45c3-9f37-b4d29091852c-istio-podinfo" pod "744e223f-d9d1-45c3-9f37-b4d29091852c" (UID: "744e223f-d9d1-45c3-9f37-b4d29091852c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:31:56.626154 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.626001 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea5efed8-2dea-46bf-9162-55fe6a1459d5-kube-api-access-kj84j" pod "ea5efed8-2dea-46bf-9162-55fe6a1459d5" (UID: "ea5efed8-2dea-46bf-9162-55fe6a1459d5"). InnerVolumeSpecName "kube-api-access-kj84j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:56.627568 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.627541 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea5efed8-2dea-46bf-9162-55fe6a1459d5-istio-envoy" pod "ea5efed8-2dea-46bf-9162-55fe6a1459d5" (UID: "ea5efed8-2dea-46bf-9162-55fe6a1459d5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:56.627927 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.627901 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/744e223f-d9d1-45c3-9f37-b4d29091852c-istio-token" pod "744e223f-d9d1-45c3-9f37-b4d29091852c" (UID: "744e223f-d9d1-45c3-9f37-b4d29091852c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:56.720443 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.720403 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/961c219a-a80f-46eb-ab6c-2920a28419e2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/961c219a-a80f-46eb-ab6c-2920a28419e2-istio-podinfo\") pod \"961c219a-a80f-46eb-ab6c-2920a28419e2\" (UID: \"961c219a-a80f-46eb-ab6c-2920a28419e2\") " Oct 30 22:31:56.720638 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.720456 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/961c219a-a80f-46eb-ab6c-2920a28419e2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/961c219a-a80f-46eb-ab6c-2920a28419e2-istio-data\") pod \"961c219a-a80f-46eb-ab6c-2920a28419e2\" (UID: \"961c219a-a80f-46eb-ab6c-2920a28419e2\") " Oct 30 22:31:56.720638 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.720501 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/961c219a-a80f-46eb-ab6c-2920a28419e2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/961c219a-a80f-46eb-ab6c-2920a28419e2-istiod-ca-cert\") pod \"961c219a-a80f-46eb-ab6c-2920a28419e2\" (UID: \"961c219a-a80f-46eb-ab6c-2920a28419e2\") " Oct 30 22:31:56.720638 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.720561 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/961c219a-a80f-46eb-ab6c-2920a28419e2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/961c219a-a80f-46eb-ab6c-2920a28419e2-workload-certs\") pod \"961c219a-a80f-46eb-ab6c-2920a28419e2\" (UID: \"961c219a-a80f-46eb-ab6c-2920a28419e2\") " Oct 30 22:31:56.720638 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.720606 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/961c219a-a80f-46eb-ab6c-2920a28419e2-istio-token\" (UniqueName: \"kubernetes.io/projected/961c219a-a80f-46eb-ab6c-2920a28419e2-istio-token\") pod \"961c219a-a80f-46eb-ab6c-2920a28419e2\" (UID: \"961c219a-a80f-46eb-ab6c-2920a28419e2\") " Oct 30 22:31:56.720882 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.720659 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/961c219a-a80f-46eb-ab6c-2920a28419e2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/961c219a-a80f-46eb-ab6c-2920a28419e2-workload-socket\") pod \"961c219a-a80f-46eb-ab6c-2920a28419e2\" (UID: \"961c219a-a80f-46eb-ab6c-2920a28419e2\") " Oct 30 22:31:56.720882 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.720734 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/961c219a-a80f-46eb-ab6c-2920a28419e2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/961c219a-a80f-46eb-ab6c-2920a28419e2-credential-socket\") pod \"961c219a-a80f-46eb-ab6c-2920a28419e2\" (UID: \"961c219a-a80f-46eb-ab6c-2920a28419e2\") " Oct 30 22:31:56.720882 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.720805 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/961c219a-a80f-46eb-ab6c-2920a28419e2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/961c219a-a80f-46eb-ab6c-2920a28419e2-istio-envoy\") pod \"961c219a-a80f-46eb-ab6c-2920a28419e2\" (UID: \"961c219a-a80f-46eb-ab6c-2920a28419e2\") " Oct 30 22:31:56.720882 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.720845 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/961c219a-a80f-46eb-ab6c-2920a28419e2-istio-data" pod "961c219a-a80f-46eb-ab6c-2920a28419e2" (UID: "961c219a-a80f-46eb-ab6c-2920a28419e2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:56.720882 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.720862 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/961c219a-a80f-46eb-ab6c-2920a28419e2-kube-api-access-8f5k4\" (UniqueName: \"kubernetes.io/projected/961c219a-a80f-46eb-ab6c-2920a28419e2-kube-api-access-8f5k4\") pod \"961c219a-a80f-46eb-ab6c-2920a28419e2\" (UID: \"961c219a-a80f-46eb-ab6c-2920a28419e2\") " Oct 30 22:31:56.721149 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.720910 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/961c219a-a80f-46eb-ab6c-2920a28419e2-workload-certs" pod "961c219a-a80f-46eb-ab6c-2920a28419e2" (UID: "961c219a-a80f-46eb-ab6c-2920a28419e2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:56.721149 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.720967 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/961c219a-a80f-46eb-ab6c-2920a28419e2-istiod-ca-cert" pod "961c219a-a80f-46eb-ab6c-2920a28419e2" (UID: "961c219a-a80f-46eb-ab6c-2920a28419e2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:31:56.721279 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.721254 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/961c219a-a80f-46eb-ab6c-2920a28419e2-credential-socket" pod "961c219a-a80f-46eb-ab6c-2920a28419e2" (UID: "961c219a-a80f-46eb-ab6c-2920a28419e2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:56.721459 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.721055 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/961c219a-a80f-46eb-ab6c-2920a28419e2-workload-socket" pod "961c219a-a80f-46eb-ab6c-2920a28419e2" (UID: "961c219a-a80f-46eb-ab6c-2920a28419e2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:56.722144 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.721633 6720 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/744e223f-d9d1-45c3-9f37-b4d29091852c-istio-token\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:56.722144 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.721664 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kj84j\" (UniqueName: \"kubernetes.io/projected/ea5efed8-2dea-46bf-9162-55fe6a1459d5-kube-api-access-kj84j\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:56.722144 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.721680 6720 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/744e223f-d9d1-45c3-9f37-b4d29091852c-workload-socket\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:56.722144 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.721694 6720 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea5efed8-2dea-46bf-9162-55fe6a1459d5-istio-podinfo\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:56.722144 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.721708 6720 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/744e223f-d9d1-45c3-9f37-b4d29091852c-istiod-ca-cert\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:56.722144 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.721722 6720 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea5efed8-2dea-46bf-9162-55fe6a1459d5-workload-socket\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:56.722144 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.721736 6720 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/961c219a-a80f-46eb-ab6c-2920a28419e2-credential-socket\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:56.722144 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.721751 6720 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/744e223f-d9d1-45c3-9f37-b4d29091852c-istio-data\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:56.722144 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.721765 6720 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea5efed8-2dea-46bf-9162-55fe6a1459d5-credential-socket\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:56.722144 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.721779 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnk7n\" (UniqueName: \"kubernetes.io/projected/744e223f-d9d1-45c3-9f37-b4d29091852c-kube-api-access-wnk7n\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:56.722144 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.721792 6720 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/744e223f-d9d1-45c3-9f37-b4d29091852c-istio-envoy\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:56.722144 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.721805 6720 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea5efed8-2dea-46bf-9162-55fe6a1459d5-istio-token\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:56.722144 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.721819 6720 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/744e223f-d9d1-45c3-9f37-b4d29091852c-istio-podinfo\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:56.722144 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.721833 6720 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/961c219a-a80f-46eb-ab6c-2920a28419e2-istio-data\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:56.722144 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.721913 6720 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea5efed8-2dea-46bf-9162-55fe6a1459d5-istio-envoy\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:56.722144 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.722024 6720 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea5efed8-2dea-46bf-9162-55fe6a1459d5-istio-data\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:56.722144 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.722038 6720 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/961c219a-a80f-46eb-ab6c-2920a28419e2-istiod-ca-cert\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:56.722144 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.722051 6720 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea5efed8-2dea-46bf-9162-55fe6a1459d5-workload-certs\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:56.722144 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.722065 6720 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/744e223f-d9d1-45c3-9f37-b4d29091852c-workload-certs\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:56.722144 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.722080 6720 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea5efed8-2dea-46bf-9162-55fe6a1459d5-istiod-ca-cert\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:56.722144 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.722095 6720 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/744e223f-d9d1-45c3-9f37-b4d29091852c-credential-socket\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:56.722144 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.722109 6720 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/961c219a-a80f-46eb-ab6c-2920a28419e2-workload-certs\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:56.723427 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.723293 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/961c219a-a80f-46eb-ab6c-2920a28419e2-istio-podinfo" pod "961c219a-a80f-46eb-ab6c-2920a28419e2" (UID: "961c219a-a80f-46eb-ab6c-2920a28419e2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:31:56.723427 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.723334 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/961c219a-a80f-46eb-ab6c-2920a28419e2-kube-api-access-8f5k4" pod "961c219a-a80f-46eb-ab6c-2920a28419e2" (UID: "961c219a-a80f-46eb-ab6c-2920a28419e2"). InnerVolumeSpecName "kube-api-access-8f5k4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:56.723568 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.723549 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/961c219a-a80f-46eb-ab6c-2920a28419e2-istio-token" pod "961c219a-a80f-46eb-ab6c-2920a28419e2" (UID: "961c219a-a80f-46eb-ab6c-2920a28419e2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:31:56.723739 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.723724 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/961c219a-a80f-46eb-ab6c-2920a28419e2-istio-envoy" pod "961c219a-a80f-46eb-ab6c-2920a28419e2" (UID: "961c219a-a80f-46eb-ab6c-2920a28419e2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:31:56.823946 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.823865 6720 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/961c219a-a80f-46eb-ab6c-2920a28419e2-istio-podinfo\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:56.823946 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.823894 6720 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/961c219a-a80f-46eb-ab6c-2920a28419e2-istio-token\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:56.823946 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.823904 6720 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/961c219a-a80f-46eb-ab6c-2920a28419e2-workload-socket\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:56.823946 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.823912 6720 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/961c219a-a80f-46eb-ab6c-2920a28419e2-istio-envoy\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:56.823946 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.823921 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8f5k4\" (UniqueName: \"kubernetes.io/projected/961c219a-a80f-46eb-ab6c-2920a28419e2-kube-api-access-8f5k4\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:31:56.934124 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.934087 6720 generic.go:356] "Generic (PLEG): container finished" podID="961c219a-a80f-46eb-ab6c-2920a28419e2" containerID="093ba049dfdd3a8f1194b04b2e1b09bc49974a3d64ca79369d6482d8f22859ce" exitCode=0 Oct 30 22:31:56.934704 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.934146 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-vnctc" Oct 30 22:31:56.934704 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.934151 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-vnctc" event={"ID":"961c219a-a80f-46eb-ab6c-2920a28419e2","Type":"ContainerDied","Data":"093ba049dfdd3a8f1194b04b2e1b09bc49974a3d64ca79369d6482d8f22859ce"} Oct 30 22:31:56.934704 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.934252 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-vnctc" event={"ID":"961c219a-a80f-46eb-ab6c-2920a28419e2","Type":"ContainerDied","Data":"a99c17bfa741c253e12b2a7741e282bce2da05b13978aeec1e82cb7cc69437f1"} Oct 30 22:31:56.934704 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.934287 6720 scope.go:122] "RemoveContainer" containerID="093ba049dfdd3a8f1194b04b2e1b09bc49974a3d64ca79369d6482d8f22859ce" Oct 30 22:31:56.936706 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.936577 6720 generic.go:356] "Generic (PLEG): container finished" podID="ea5efed8-2dea-46bf-9162-55fe6a1459d5" containerID="186fdb74a0368dc7020f7e17c2afefa3e66b060f9f47ec8464d20ee09504852c" exitCode=0 Oct 30 22:31:56.936706 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.936608 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-x64z9" event={"ID":"ea5efed8-2dea-46bf-9162-55fe6a1459d5","Type":"ContainerDied","Data":"186fdb74a0368dc7020f7e17c2afefa3e66b060f9f47ec8464d20ee09504852c"} Oct 30 22:31:56.936706 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.936666 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-x64z9" Oct 30 22:31:56.936926 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.936667 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-x64z9" event={"ID":"ea5efed8-2dea-46bf-9162-55fe6a1459d5","Type":"ContainerDied","Data":"6c3e80a149f0c385a74fc7a52127be22089f365b8b331d5a3c233c54703f6d8d"} Oct 30 22:31:56.938316 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.938290 6720 generic.go:356] "Generic (PLEG): container finished" podID="744e223f-d9d1-45c3-9f37-b4d29091852c" containerID="bedcecf53043ca5387632a0ffd82fb1bb5179d55f20c8e358e8bf667d5e3237d" exitCode=0 Oct 30 22:31:56.938504 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.938350 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-v4kbr" Oct 30 22:31:56.938504 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.938492 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-v4kbr" event={"ID":"744e223f-d9d1-45c3-9f37-b4d29091852c","Type":"ContainerDied","Data":"bedcecf53043ca5387632a0ffd82fb1bb5179d55f20c8e358e8bf667d5e3237d"} Oct 30 22:31:56.938624 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.938526 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-v4kbr" event={"ID":"744e223f-d9d1-45c3-9f37-b4d29091852c","Type":"ContainerDied","Data":"0bd8296e97511290b70a3112c62686d2a003628a96bec68b8b0d2766cda04a13"} Oct 30 22:31:56.948300 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.948246 6720 scope.go:122] "RemoveContainer" containerID="093ba049dfdd3a8f1194b04b2e1b09bc49974a3d64ca79369d6482d8f22859ce" Oct 30 22:31:56.948736 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:31:56.948711 6720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"093ba049dfdd3a8f1194b04b2e1b09bc49974a3d64ca79369d6482d8f22859ce\": container with ID starting with 093ba049dfdd3a8f1194b04b2e1b09bc49974a3d64ca79369d6482d8f22859ce not found: ID does not exist" containerID="093ba049dfdd3a8f1194b04b2e1b09bc49974a3d64ca79369d6482d8f22859ce" Oct 30 22:31:56.948853 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.948828 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"093ba049dfdd3a8f1194b04b2e1b09bc49974a3d64ca79369d6482d8f22859ce"} err="failed to get container status \"093ba049dfdd3a8f1194b04b2e1b09bc49974a3d64ca79369d6482d8f22859ce\": rpc error: code = NotFound desc = could not find container \"093ba049dfdd3a8f1194b04b2e1b09bc49974a3d64ca79369d6482d8f22859ce\": container with ID starting with 093ba049dfdd3a8f1194b04b2e1b09bc49974a3d64ca79369d6482d8f22859ce not found: ID does not exist" Oct 30 22:31:56.948983 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.948857 6720 scope.go:122] "RemoveContainer" containerID="186fdb74a0368dc7020f7e17c2afefa3e66b060f9f47ec8464d20ee09504852c" Oct 30 22:31:56.958940 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.958920 6720 scope.go:122] "RemoveContainer" containerID="186fdb74a0368dc7020f7e17c2afefa3e66b060f9f47ec8464d20ee09504852c" Oct 30 22:31:56.959209 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:31:56.959191 6720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"186fdb74a0368dc7020f7e17c2afefa3e66b060f9f47ec8464d20ee09504852c\": container with ID starting with 186fdb74a0368dc7020f7e17c2afefa3e66b060f9f47ec8464d20ee09504852c not found: ID does not exist" containerID="186fdb74a0368dc7020f7e17c2afefa3e66b060f9f47ec8464d20ee09504852c" Oct 30 22:31:56.959263 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.959215 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"186fdb74a0368dc7020f7e17c2afefa3e66b060f9f47ec8464d20ee09504852c"} err="failed to get container status \"186fdb74a0368dc7020f7e17c2afefa3e66b060f9f47ec8464d20ee09504852c\": rpc error: code = NotFound desc = could not find container \"186fdb74a0368dc7020f7e17c2afefa3e66b060f9f47ec8464d20ee09504852c\": container with ID starting with 186fdb74a0368dc7020f7e17c2afefa3e66b060f9f47ec8464d20ee09504852c not found: ID does not exist" Oct 30 22:31:56.959263 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.959233 6720 scope.go:122] "RemoveContainer" containerID="bedcecf53043ca5387632a0ffd82fb1bb5179d55f20c8e358e8bf667d5e3237d" Oct 30 22:31:56.965834 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.965816 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-vnctc"] Oct 30 22:31:56.968593 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.968572 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-vnctc"] Oct 30 22:31:56.970611 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.970597 6720 scope.go:122] "RemoveContainer" containerID="bedcecf53043ca5387632a0ffd82fb1bb5179d55f20c8e358e8bf667d5e3237d" Oct 30 22:31:56.970902 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:31:56.970885 6720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bedcecf53043ca5387632a0ffd82fb1bb5179d55f20c8e358e8bf667d5e3237d\": container with ID starting with bedcecf53043ca5387632a0ffd82fb1bb5179d55f20c8e358e8bf667d5e3237d not found: ID does not exist" containerID="bedcecf53043ca5387632a0ffd82fb1bb5179d55f20c8e358e8bf667d5e3237d" Oct 30 22:31:56.970942 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.970913 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bedcecf53043ca5387632a0ffd82fb1bb5179d55f20c8e358e8bf667d5e3237d"} err="failed to get container status \"bedcecf53043ca5387632a0ffd82fb1bb5179d55f20c8e358e8bf667d5e3237d\": rpc error: code = NotFound desc = could not find container \"bedcecf53043ca5387632a0ffd82fb1bb5179d55f20c8e358e8bf667d5e3237d\": container with ID starting with bedcecf53043ca5387632a0ffd82fb1bb5179d55f20c8e358e8bf667d5e3237d not found: ID does not exist" Oct 30 22:31:56.979111 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.979091 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-x64z9"] Oct 30 22:31:56.985607 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:56.985588 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-x64z9"] Oct 30 22:31:57.001245 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:57.001225 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-v4kbr"] Oct 30 22:31:57.008997 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:57.008975 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-120cb1-openshift-default-d97c644bf-v4kbr"] Oct 30 22:31:57.424167 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:57.418361 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:57.958668 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:57.958626 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="744e223f-d9d1-45c3-9f37-b4d29091852c" path="/var/lib/kubelet/pods/744e223f-d9d1-45c3-9f37-b4d29091852c/volumes" Oct 30 22:31:57.959182 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:57.959165 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="961c219a-a80f-46eb-ab6c-2920a28419e2" path="/var/lib/kubelet/pods/961c219a-a80f-46eb-ab6c-2920a28419e2/volumes" Oct 30 22:31:57.959710 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:57.959692 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea5efed8-2dea-46bf-9162-55fe6a1459d5" path="/var/lib/kubelet/pods/ea5efed8-2dea-46bf-9162-55fe6a1459d5/volumes" Oct 30 22:31:58.218107 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:58.217896 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:59.015425 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:59.009767 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:59.169780 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:59.169750 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:59.320355 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:59.320270 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:59.518675 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:59.518638 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:31:59.840470 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:31:59.837788 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:00.185467 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:00.185427 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jqtlj"] Oct 30 22:32:00.185928 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:00.185781 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jqtlj" podUID="fc5329cd-398f-42a3-8e4d-82d3b6446773" containerName="istio-proxy" containerID="cri-o://2919ba0d0313b3eb597370b898c04f77e02a181debed9e2cd4adc12b32a9fd3e" gracePeriod=30 Oct 30 22:32:00.794475 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:00.793705 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:00.942217 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:00.942187 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:01.439614 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:01.439577 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:01.912955 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:01.912922 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:02.052191 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:02.052151 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:02.226005 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:02.225912 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:02.999259 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:02.999228 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:03.752281 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:03.752244 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:04.389656 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:04.389536 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:05.041125 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:05.041093 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:05.042088 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:05.042055 6720 patch_prober.go:29] interesting pod/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jqtlj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:32:05.042239 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:05.042110 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jqtlj" podUID="fc5329cd-398f-42a3-8e4d-82d3b6446773" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:32:05.457763 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:05.457740 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jqtlj" Oct 30 22:32:05.544550 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:05.544519 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc5329cd-398f-42a3-8e4d-82d3b6446773-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc5329cd-398f-42a3-8e4d-82d3b6446773-workload-socket\") pod \"fc5329cd-398f-42a3-8e4d-82d3b6446773\" (UID: \"fc5329cd-398f-42a3-8e4d-82d3b6446773\") " Oct 30 22:32:05.544723 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:05.544570 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc5329cd-398f-42a3-8e4d-82d3b6446773-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc5329cd-398f-42a3-8e4d-82d3b6446773-credential-socket\") pod \"fc5329cd-398f-42a3-8e4d-82d3b6446773\" (UID: \"fc5329cd-398f-42a3-8e4d-82d3b6446773\") " Oct 30 22:32:05.544723 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:05.544618 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc5329cd-398f-42a3-8e4d-82d3b6446773-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc5329cd-398f-42a3-8e4d-82d3b6446773-istio-envoy\") pod \"fc5329cd-398f-42a3-8e4d-82d3b6446773\" (UID: \"fc5329cd-398f-42a3-8e4d-82d3b6446773\") " Oct 30 22:32:05.544723 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:05.544713 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc5329cd-398f-42a3-8e4d-82d3b6446773-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc5329cd-398f-42a3-8e4d-82d3b6446773-istiod-ca-cert\") pod \"fc5329cd-398f-42a3-8e4d-82d3b6446773\" (UID: \"fc5329cd-398f-42a3-8e4d-82d3b6446773\") " Oct 30 22:32:05.544897 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:05.544760 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc5329cd-398f-42a3-8e4d-82d3b6446773-istio-token\" (UniqueName: \"kubernetes.io/projected/fc5329cd-398f-42a3-8e4d-82d3b6446773-istio-token\") pod \"fc5329cd-398f-42a3-8e4d-82d3b6446773\" (UID: \"fc5329cd-398f-42a3-8e4d-82d3b6446773\") " Oct 30 22:32:05.544897 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:05.544797 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc5329cd-398f-42a3-8e4d-82d3b6446773-workload-socket" pod "fc5329cd-398f-42a3-8e4d-82d3b6446773" (UID: "fc5329cd-398f-42a3-8e4d-82d3b6446773"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:32:05.544897 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:05.544825 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc5329cd-398f-42a3-8e4d-82d3b6446773-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc5329cd-398f-42a3-8e4d-82d3b6446773-istio-data\") pod \"fc5329cd-398f-42a3-8e4d-82d3b6446773\" (UID: \"fc5329cd-398f-42a3-8e4d-82d3b6446773\") " Oct 30 22:32:05.544897 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:05.544876 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc5329cd-398f-42a3-8e4d-82d3b6446773-credential-socket" pod "fc5329cd-398f-42a3-8e4d-82d3b6446773" (UID: "fc5329cd-398f-42a3-8e4d-82d3b6446773"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:32:05.545110 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:05.544933 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc5329cd-398f-42a3-8e4d-82d3b6446773-kube-api-access-stws9\" (UniqueName: \"kubernetes.io/projected/fc5329cd-398f-42a3-8e4d-82d3b6446773-kube-api-access-stws9\") pod \"fc5329cd-398f-42a3-8e4d-82d3b6446773\" (UID: \"fc5329cd-398f-42a3-8e4d-82d3b6446773\") " Oct 30 22:32:05.545110 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:05.544975 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc5329cd-398f-42a3-8e4d-82d3b6446773-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc5329cd-398f-42a3-8e4d-82d3b6446773-workload-certs\") pod \"fc5329cd-398f-42a3-8e4d-82d3b6446773\" (UID: \"fc5329cd-398f-42a3-8e4d-82d3b6446773\") " Oct 30 22:32:05.545110 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:05.545027 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fc5329cd-398f-42a3-8e4d-82d3b6446773-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc5329cd-398f-42a3-8e4d-82d3b6446773-istio-podinfo\") pod \"fc5329cd-398f-42a3-8e4d-82d3b6446773\" (UID: \"fc5329cd-398f-42a3-8e4d-82d3b6446773\") " Oct 30 22:32:05.545110 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:05.545071 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc5329cd-398f-42a3-8e4d-82d3b6446773-istio-data" pod "fc5329cd-398f-42a3-8e4d-82d3b6446773" (UID: "fc5329cd-398f-42a3-8e4d-82d3b6446773"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:32:05.545340 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:05.545155 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc5329cd-398f-42a3-8e4d-82d3b6446773-istiod-ca-cert" pod "fc5329cd-398f-42a3-8e4d-82d3b6446773" (UID: "fc5329cd-398f-42a3-8e4d-82d3b6446773"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:32:05.545340 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:05.545315 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc5329cd-398f-42a3-8e4d-82d3b6446773-workload-certs" pod "fc5329cd-398f-42a3-8e4d-82d3b6446773" (UID: "fc5329cd-398f-42a3-8e4d-82d3b6446773"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:32:05.545937 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:05.545911 6720 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc5329cd-398f-42a3-8e4d-82d3b6446773-istio-data\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:05.546042 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:05.545940 6720 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc5329cd-398f-42a3-8e4d-82d3b6446773-workload-certs\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:05.546042 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:05.545958 6720 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc5329cd-398f-42a3-8e4d-82d3b6446773-workload-socket\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:05.546042 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:05.545972 6720 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc5329cd-398f-42a3-8e4d-82d3b6446773-credential-socket\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:05.546042 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:05.545987 6720 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc5329cd-398f-42a3-8e4d-82d3b6446773-istiod-ca-cert\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:05.547222 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:05.547194 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc5329cd-398f-42a3-8e4d-82d3b6446773-istio-envoy" pod "fc5329cd-398f-42a3-8e4d-82d3b6446773" (UID: "fc5329cd-398f-42a3-8e4d-82d3b6446773"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:32:05.547335 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:05.547284 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fc5329cd-398f-42a3-8e4d-82d3b6446773-istio-podinfo" pod "fc5329cd-398f-42a3-8e4d-82d3b6446773" (UID: "fc5329cd-398f-42a3-8e4d-82d3b6446773"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:32:05.547396 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:05.547352 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc5329cd-398f-42a3-8e4d-82d3b6446773-kube-api-access-stws9" pod "fc5329cd-398f-42a3-8e4d-82d3b6446773" (UID: "fc5329cd-398f-42a3-8e4d-82d3b6446773"). InnerVolumeSpecName "kube-api-access-stws9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:32:05.547460 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:05.547372 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc5329cd-398f-42a3-8e4d-82d3b6446773-istio-token" pod "fc5329cd-398f-42a3-8e4d-82d3b6446773" (UID: "fc5329cd-398f-42a3-8e4d-82d3b6446773"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:32:05.646683 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:05.646654 6720 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc5329cd-398f-42a3-8e4d-82d3b6446773-istio-envoy\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:05.646683 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:05.646679 6720 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc5329cd-398f-42a3-8e4d-82d3b6446773-istio-token\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:05.646683 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:05.646689 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stws9\" (UniqueName: \"kubernetes.io/projected/fc5329cd-398f-42a3-8e4d-82d3b6446773-kube-api-access-stws9\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:05.646923 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:05.646698 6720 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc5329cd-398f-42a3-8e4d-82d3b6446773-istio-podinfo\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:05.824187 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:05.815459 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:06.010116 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:06.010081 6720 generic.go:356] "Generic (PLEG): container finished" podID="fc5329cd-398f-42a3-8e4d-82d3b6446773" containerID="2919ba0d0313b3eb597370b898c04f77e02a181debed9e2cd4adc12b32a9fd3e" exitCode=0 Oct 30 22:32:06.010282 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:06.010134 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jqtlj" event={"ID":"fc5329cd-398f-42a3-8e4d-82d3b6446773","Type":"ContainerDied","Data":"2919ba0d0313b3eb597370b898c04f77e02a181debed9e2cd4adc12b32a9fd3e"} Oct 30 22:32:06.010282 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:06.010193 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jqtlj" event={"ID":"fc5329cd-398f-42a3-8e4d-82d3b6446773","Type":"ContainerDied","Data":"3cabc6e44e7f2e2f41bdea1abfd3483ef86fd0686a683c850084257f7ad79f7f"} Oct 30 22:32:06.010282 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:06.010213 6720 scope.go:122] "RemoveContainer" containerID="2919ba0d0313b3eb597370b898c04f77e02a181debed9e2cd4adc12b32a9fd3e" Oct 30 22:32:06.010282 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:06.010135 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jqtlj" Oct 30 22:32:06.020761 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:06.020744 6720 scope.go:122] "RemoveContainer" containerID="2919ba0d0313b3eb597370b898c04f77e02a181debed9e2cd4adc12b32a9fd3e" Oct 30 22:32:06.021000 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:32:06.020981 6720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2919ba0d0313b3eb597370b898c04f77e02a181debed9e2cd4adc12b32a9fd3e\": container with ID starting with 2919ba0d0313b3eb597370b898c04f77e02a181debed9e2cd4adc12b32a9fd3e not found: ID does not exist" containerID="2919ba0d0313b3eb597370b898c04f77e02a181debed9e2cd4adc12b32a9fd3e" Oct 30 22:32:06.021041 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:06.021009 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2919ba0d0313b3eb597370b898c04f77e02a181debed9e2cd4adc12b32a9fd3e"} err="failed to get container status \"2919ba0d0313b3eb597370b898c04f77e02a181debed9e2cd4adc12b32a9fd3e\": rpc error: code = NotFound desc = could not find container \"2919ba0d0313b3eb597370b898c04f77e02a181debed9e2cd4adc12b32a9fd3e\": container with ID starting with 2919ba0d0313b3eb597370b898c04f77e02a181debed9e2cd4adc12b32a9fd3e not found: ID does not exist" Oct 30 22:32:06.031889 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:06.031868 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jqtlj"] Oct 30 22:32:06.036531 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:06.036510 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-0f26c5-openshift-default-5f494fb979-jqtlj"] Oct 30 22:32:06.443687 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:06.443622 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:06.696091 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:06.696021 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:06.871528 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:06.871488 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:06.965226 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:06.965134 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4b2bb0-ffd47c566-qwcvz"] Oct 30 22:32:06.965498 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:06.965468 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b2bb0-ffd47c566-qwcvz" podUID="6b6fc7b5-8eca-4f2c-b012-6f9375211314" containerName="maas-api" containerID="cri-o://b670497784a390c55f50c438626e2e7c0e95b76477655655ee157d3527ec12fd" gracePeriod=30 Oct 30 22:32:07.022991 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:07.022919 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:07.195414 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:07.195233 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:07.348930 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:07.348881 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:07.415568 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:07.415543 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b2bb0-ffd47c566-qwcvz" Oct 30 22:32:07.487878 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:07.487795 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b6fc7b5-8eca-4f2c-b012-6f9375211314-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b6fc7b5-8eca-4f2c-b012-6f9375211314-maas-api-tls\") pod \"6b6fc7b5-8eca-4f2c-b012-6f9375211314\" (UID: \"6b6fc7b5-8eca-4f2c-b012-6f9375211314\") " Oct 30 22:32:07.487878 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:07.487836 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b6fc7b5-8eca-4f2c-b012-6f9375211314-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b6fc7b5-8eca-4f2c-b012-6f9375211314-metrics-certs\") pod \"6b6fc7b5-8eca-4f2c-b012-6f9375211314\" (UID: \"6b6fc7b5-8eca-4f2c-b012-6f9375211314\") " Oct 30 22:32:07.488127 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:07.487954 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b6fc7b5-8eca-4f2c-b012-6f9375211314-kube-api-access-sxq5v\" (UniqueName: \"kubernetes.io/projected/6b6fc7b5-8eca-4f2c-b012-6f9375211314-kube-api-access-sxq5v\") pod \"6b6fc7b5-8eca-4f2c-b012-6f9375211314\" (UID: \"6b6fc7b5-8eca-4f2c-b012-6f9375211314\") " Oct 30 22:32:07.488127 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:07.488082 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b6fc7b5-8eca-4f2c-b012-6f9375211314-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b6fc7b5-8eca-4f2c-b012-6f9375211314-openshift-service-ca\") pod \"6b6fc7b5-8eca-4f2c-b012-6f9375211314\" (UID: \"6b6fc7b5-8eca-4f2c-b012-6f9375211314\") " Oct 30 22:32:07.488683 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:07.488644 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b6fc7b5-8eca-4f2c-b012-6f9375211314-openshift-service-ca" pod "6b6fc7b5-8eca-4f2c-b012-6f9375211314" (UID: "6b6fc7b5-8eca-4f2c-b012-6f9375211314"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:32:07.492001 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:07.490779 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b6fc7b5-8eca-4f2c-b012-6f9375211314-metrics-certs" pod "6b6fc7b5-8eca-4f2c-b012-6f9375211314" (UID: "6b6fc7b5-8eca-4f2c-b012-6f9375211314"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:32:07.492001 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:07.490827 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b6fc7b5-8eca-4f2c-b012-6f9375211314-kube-api-access-sxq5v" pod "6b6fc7b5-8eca-4f2c-b012-6f9375211314" (UID: "6b6fc7b5-8eca-4f2c-b012-6f9375211314"). InnerVolumeSpecName "kube-api-access-sxq5v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:32:07.492709 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:07.492562 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b6fc7b5-8eca-4f2c-b012-6f9375211314-maas-api-tls" pod "6b6fc7b5-8eca-4f2c-b012-6f9375211314" (UID: "6b6fc7b5-8eca-4f2c-b012-6f9375211314"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:32:07.500146 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:07.500124 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:07.590129 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:07.590085 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxq5v\" (UniqueName: \"kubernetes.io/projected/6b6fc7b5-8eca-4f2c-b012-6f9375211314-kube-api-access-sxq5v\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:07.590129 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:07.590118 6720 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b6fc7b5-8eca-4f2c-b012-6f9375211314-openshift-service-ca\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:07.590129 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:07.590131 6720 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b6fc7b5-8eca-4f2c-b012-6f9375211314-maas-api-tls\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:07.590129 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:07.590140 6720 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b6fc7b5-8eca-4f2c-b012-6f9375211314-metrics-certs\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:07.797222 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:07.797137 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:07.957658 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:07.957624 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc5329cd-398f-42a3-8e4d-82d3b6446773" path="/var/lib/kubelet/pods/fc5329cd-398f-42a3-8e4d-82d3b6446773/volumes" Oct 30 22:32:08.037500 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:08.037461 6720 generic.go:356] "Generic (PLEG): container finished" podID="6b6fc7b5-8eca-4f2c-b012-6f9375211314" containerID="b670497784a390c55f50c438626e2e7c0e95b76477655655ee157d3527ec12fd" exitCode=0 Oct 30 22:32:08.037679 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:08.037505 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b2bb0-ffd47c566-qwcvz" Oct 30 22:32:08.037679 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:08.037537 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b2bb0-ffd47c566-qwcvz" event={"ID":"6b6fc7b5-8eca-4f2c-b012-6f9375211314","Type":"ContainerDied","Data":"b670497784a390c55f50c438626e2e7c0e95b76477655655ee157d3527ec12fd"} Oct 30 22:32:08.037679 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:08.037589 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-4b2bb0-ffd47c566-qwcvz" event={"ID":"6b6fc7b5-8eca-4f2c-b012-6f9375211314","Type":"ContainerDied","Data":"06178a2e194c31f8164770a03bbab663c8a0a8dd6b4ef78226861ebe57e4e00c"} Oct 30 22:32:08.037679 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:08.037613 6720 scope.go:122] "RemoveContainer" containerID="b670497784a390c55f50c438626e2e7c0e95b76477655655ee157d3527ec12fd" Oct 30 22:32:08.052577 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:08.052550 6720 scope.go:122] "RemoveContainer" containerID="b670497784a390c55f50c438626e2e7c0e95b76477655655ee157d3527ec12fd" Oct 30 22:32:08.053127 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:32:08.053104 6720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b670497784a390c55f50c438626e2e7c0e95b76477655655ee157d3527ec12fd\": container with ID starting with b670497784a390c55f50c438626e2e7c0e95b76477655655ee157d3527ec12fd not found: ID does not exist" containerID="b670497784a390c55f50c438626e2e7c0e95b76477655655ee157d3527ec12fd" Oct 30 22:32:08.053246 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:08.053134 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b670497784a390c55f50c438626e2e7c0e95b76477655655ee157d3527ec12fd"} err="failed to get container status \"b670497784a390c55f50c438626e2e7c0e95b76477655655ee157d3527ec12fd\": rpc error: code = NotFound desc = could not find container \"b670497784a390c55f50c438626e2e7c0e95b76477655655ee157d3527ec12fd\": container with ID starting with b670497784a390c55f50c438626e2e7c0e95b76477655655ee157d3527ec12fd not found: ID does not exist" Oct 30 22:32:08.058495 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:08.058475 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4b2bb0-ffd47c566-qwcvz"] Oct 30 22:32:08.062460 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:08.062440 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-4b2bb0-ffd47c566-qwcvz"] Oct 30 22:32:08.285237 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:08.284741 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:08.904715 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:08.904663 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:09.544740 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:09.544703 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:09.960369 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:09.960325 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b6fc7b5-8eca-4f2c-b012-6f9375211314" path="/var/lib/kubelet/pods/6b6fc7b5-8eca-4f2c-b012-6f9375211314/volumes" Oct 30 22:32:10.186296 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:10.186260 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:10.325578 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:10.325482 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:10.454352 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:32:10.454314 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 22:32:11.048233 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:11.048194 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:11.824068 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:11.824030 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:12.518545 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:12.518512 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-1a5bdf-7d9c8596d4-lvfnt"] Oct 30 22:32:12.519071 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:12.518771 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1a5bdf-7d9c8596d4-lvfnt" podUID="5267ee0d-42b1-482f-aa8c-f1cba3b748ea" containerName="maas-api" containerID="cri-o://49b2cf4fad8c8bb29da368f7fbbc3a83a4205b52c0b3740b2a1607b6e97139e4" gracePeriod=30 Oct 30 22:32:12.610621 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:12.610570 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:12.794117 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:12.793875 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:12.951464 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:12.951436 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:12.970049 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:12.970022 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1a5bdf-7d9c8596d4-lvfnt" Oct 30 22:32:13.028183 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:13.025497 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5267ee0d-42b1-482f-aa8c-f1cba3b748ea-kube-api-access-xxmpn\" (UniqueName: \"kubernetes.io/projected/5267ee0d-42b1-482f-aa8c-f1cba3b748ea-kube-api-access-xxmpn\") pod \"5267ee0d-42b1-482f-aa8c-f1cba3b748ea\" (UID: \"5267ee0d-42b1-482f-aa8c-f1cba3b748ea\") " Oct 30 22:32:13.028183 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:13.025589 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5267ee0d-42b1-482f-aa8c-f1cba3b748ea-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5267ee0d-42b1-482f-aa8c-f1cba3b748ea-metrics-certs\") pod \"5267ee0d-42b1-482f-aa8c-f1cba3b748ea\" (UID: \"5267ee0d-42b1-482f-aa8c-f1cba3b748ea\") " Oct 30 22:32:13.028183 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:13.025683 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5267ee0d-42b1-482f-aa8c-f1cba3b748ea-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5267ee0d-42b1-482f-aa8c-f1cba3b748ea-maas-api-tls\") pod \"5267ee0d-42b1-482f-aa8c-f1cba3b748ea\" (UID: \"5267ee0d-42b1-482f-aa8c-f1cba3b748ea\") " Oct 30 22:32:13.028183 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:13.025716 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5267ee0d-42b1-482f-aa8c-f1cba3b748ea-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5267ee0d-42b1-482f-aa8c-f1cba3b748ea-openshift-service-ca\") pod \"5267ee0d-42b1-482f-aa8c-f1cba3b748ea\" (UID: \"5267ee0d-42b1-482f-aa8c-f1cba3b748ea\") " Oct 30 22:32:13.028183 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:13.027154 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5267ee0d-42b1-482f-aa8c-f1cba3b748ea-openshift-service-ca" pod "5267ee0d-42b1-482f-aa8c-f1cba3b748ea" (UID: "5267ee0d-42b1-482f-aa8c-f1cba3b748ea"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:32:13.057156 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:13.057102 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5267ee0d-42b1-482f-aa8c-f1cba3b748ea-kube-api-access-xxmpn" pod "5267ee0d-42b1-482f-aa8c-f1cba3b748ea" (UID: "5267ee0d-42b1-482f-aa8c-f1cba3b748ea"). InnerVolumeSpecName "kube-api-access-xxmpn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:32:13.058647 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:13.058616 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5267ee0d-42b1-482f-aa8c-f1cba3b748ea-maas-api-tls" pod "5267ee0d-42b1-482f-aa8c-f1cba3b748ea" (UID: "5267ee0d-42b1-482f-aa8c-f1cba3b748ea"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:32:13.058785 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:13.058738 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5267ee0d-42b1-482f-aa8c-f1cba3b748ea-metrics-certs" pod "5267ee0d-42b1-482f-aa8c-f1cba3b748ea" (UID: "5267ee0d-42b1-482f-aa8c-f1cba3b748ea"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:32:13.076355 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:13.076323 6720 generic.go:356] "Generic (PLEG): container finished" podID="5267ee0d-42b1-482f-aa8c-f1cba3b748ea" containerID="49b2cf4fad8c8bb29da368f7fbbc3a83a4205b52c0b3740b2a1607b6e97139e4" exitCode=0 Oct 30 22:32:13.076544 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:13.076411 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1a5bdf-7d9c8596d4-lvfnt" Oct 30 22:32:13.076544 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:13.076419 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1a5bdf-7d9c8596d4-lvfnt" event={"ID":"5267ee0d-42b1-482f-aa8c-f1cba3b748ea","Type":"ContainerDied","Data":"49b2cf4fad8c8bb29da368f7fbbc3a83a4205b52c0b3740b2a1607b6e97139e4"} Oct 30 22:32:13.076544 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:13.076473 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-1a5bdf-7d9c8596d4-lvfnt" event={"ID":"5267ee0d-42b1-482f-aa8c-f1cba3b748ea","Type":"ContainerDied","Data":"86e56842c20109138a0ea6b68e7bd8f8fe3d4dabd2bf253c1bb5e6ee926f8560"} Oct 30 22:32:13.076544 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:13.076495 6720 scope.go:122] "RemoveContainer" containerID="49b2cf4fad8c8bb29da368f7fbbc3a83a4205b52c0b3740b2a1607b6e97139e4" Oct 30 22:32:13.087419 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:13.087371 6720 scope.go:122] "RemoveContainer" containerID="49b2cf4fad8c8bb29da368f7fbbc3a83a4205b52c0b3740b2a1607b6e97139e4" Oct 30 22:32:13.087690 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:32:13.087672 6720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49b2cf4fad8c8bb29da368f7fbbc3a83a4205b52c0b3740b2a1607b6e97139e4\": container with ID starting with 49b2cf4fad8c8bb29da368f7fbbc3a83a4205b52c0b3740b2a1607b6e97139e4 not found: ID does not exist" containerID="49b2cf4fad8c8bb29da368f7fbbc3a83a4205b52c0b3740b2a1607b6e97139e4" Oct 30 22:32:13.087739 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:13.087700 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49b2cf4fad8c8bb29da368f7fbbc3a83a4205b52c0b3740b2a1607b6e97139e4"} err="failed to get container status \"49b2cf4fad8c8bb29da368f7fbbc3a83a4205b52c0b3740b2a1607b6e97139e4\": rpc error: code = NotFound desc = could not find container \"49b2cf4fad8c8bb29da368f7fbbc3a83a4205b52c0b3740b2a1607b6e97139e4\": container with ID starting with 49b2cf4fad8c8bb29da368f7fbbc3a83a4205b52c0b3740b2a1607b6e97139e4 not found: ID does not exist" Oct 30 22:32:13.095369 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:13.095350 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:13.100455 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:13.100434 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-1a5bdf-7d9c8596d4-lvfnt"] Oct 30 22:32:13.103588 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:13.103567 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-1a5bdf-7d9c8596d4-lvfnt"] Oct 30 22:32:13.127824 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:13.127779 6720 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5267ee0d-42b1-482f-aa8c-f1cba3b748ea-maas-api-tls\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:13.127824 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:13.127810 6720 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5267ee0d-42b1-482f-aa8c-f1cba3b748ea-openshift-service-ca\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:13.127824 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:13.127827 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxmpn\" (UniqueName: \"kubernetes.io/projected/5267ee0d-42b1-482f-aa8c-f1cba3b748ea-kube-api-access-xxmpn\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:13.128068 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:13.127842 6720 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5267ee0d-42b1-482f-aa8c-f1cba3b748ea-metrics-certs\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:13.341048 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:13.340963 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:13.819842 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:13.819807 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:13.956338 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:13.956307 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5267ee0d-42b1-482f-aa8c-f1cba3b748ea" path="/var/lib/kubelet/pods/5267ee0d-42b1-482f-aa8c-f1cba3b748ea/volumes" Oct 30 22:32:14.510044 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:14.510010 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:14.663222 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:14.659236 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:14.974957 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:14.974920 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:15.421193 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:15.420975 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:16.225432 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:16.225322 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:16.774042 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:16.774008 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-bacea0-58688dc49f-m4cv7"] Oct 30 22:32:16.774308 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:16.774243 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bacea0-58688dc49f-m4cv7" podUID="132ce48a-6190-4f71-8b77-1a28c917030d" containerName="maas-api" containerID="cri-o://e23c19265f41345552b83a4b4160b4d096b58b4e126e3777ca42e8f6497dbfee" gracePeriod=30 Oct 30 22:32:16.858923 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:16.858893 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:17.094468 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:17.081009 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:17.166727 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:17.166663 6720 generic.go:356] "Generic (PLEG): container finished" podID="132ce48a-6190-4f71-8b77-1a28c917030d" containerID="e23c19265f41345552b83a4b4160b4d096b58b4e126e3777ca42e8f6497dbfee" exitCode=0 Oct 30 22:32:17.166916 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:17.166718 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bacea0-58688dc49f-m4cv7" event={"ID":"132ce48a-6190-4f71-8b77-1a28c917030d","Type":"ContainerDied","Data":"e23c19265f41345552b83a4b4160b4d096b58b4e126e3777ca42e8f6497dbfee"} Oct 30 22:32:17.241023 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:17.240994 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:17.257652 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:17.257625 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bacea0-58688dc49f-m4cv7" Oct 30 22:32:17.334739 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:17.334697 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/132ce48a-6190-4f71-8b77-1a28c917030d-kube-api-access-xnk5k\" (UniqueName: \"kubernetes.io/projected/132ce48a-6190-4f71-8b77-1a28c917030d-kube-api-access-xnk5k\") pod \"132ce48a-6190-4f71-8b77-1a28c917030d\" (UID: \"132ce48a-6190-4f71-8b77-1a28c917030d\") " Oct 30 22:32:17.334912 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:17.334831 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/132ce48a-6190-4f71-8b77-1a28c917030d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/132ce48a-6190-4f71-8b77-1a28c917030d-openshift-service-ca\") pod \"132ce48a-6190-4f71-8b77-1a28c917030d\" (UID: \"132ce48a-6190-4f71-8b77-1a28c917030d\") " Oct 30 22:32:17.335038 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:17.334959 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/132ce48a-6190-4f71-8b77-1a28c917030d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/132ce48a-6190-4f71-8b77-1a28c917030d-maas-api-tls\") pod \"132ce48a-6190-4f71-8b77-1a28c917030d\" (UID: \"132ce48a-6190-4f71-8b77-1a28c917030d\") " Oct 30 22:32:17.335128 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:17.335111 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/132ce48a-6190-4f71-8b77-1a28c917030d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/132ce48a-6190-4f71-8b77-1a28c917030d-metrics-certs\") pod \"132ce48a-6190-4f71-8b77-1a28c917030d\" (UID: \"132ce48a-6190-4f71-8b77-1a28c917030d\") " Oct 30 22:32:17.339759 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:17.339730 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/132ce48a-6190-4f71-8b77-1a28c917030d-openshift-service-ca" pod "132ce48a-6190-4f71-8b77-1a28c917030d" (UID: "132ce48a-6190-4f71-8b77-1a28c917030d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:32:17.342241 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:17.340676 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/132ce48a-6190-4f71-8b77-1a28c917030d-metrics-certs" pod "132ce48a-6190-4f71-8b77-1a28c917030d" (UID: "132ce48a-6190-4f71-8b77-1a28c917030d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:32:17.344219 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:17.344191 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/132ce48a-6190-4f71-8b77-1a28c917030d-kube-api-access-xnk5k" pod "132ce48a-6190-4f71-8b77-1a28c917030d" (UID: "132ce48a-6190-4f71-8b77-1a28c917030d"). InnerVolumeSpecName "kube-api-access-xnk5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:32:17.344335 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:17.344318 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/132ce48a-6190-4f71-8b77-1a28c917030d-maas-api-tls" pod "132ce48a-6190-4f71-8b77-1a28c917030d" (UID: "132ce48a-6190-4f71-8b77-1a28c917030d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:32:17.390875 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:17.390838 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:17.438271 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:17.438234 6720 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/132ce48a-6190-4f71-8b77-1a28c917030d-openshift-service-ca\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:17.438271 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:17.438270 6720 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/132ce48a-6190-4f71-8b77-1a28c917030d-maas-api-tls\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:17.438575 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:17.438282 6720 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/132ce48a-6190-4f71-8b77-1a28c917030d-metrics-certs\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:17.438575 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:17.438291 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnk5k\" (UniqueName: \"kubernetes.io/projected/132ce48a-6190-4f71-8b77-1a28c917030d-kube-api-access-xnk5k\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:17.696458 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:17.696426 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:18.054698 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:18.054651 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:18.174978 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:18.174946 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bacea0-58688dc49f-m4cv7" Oct 30 22:32:18.174978 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:18.174960 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-bacea0-58688dc49f-m4cv7" event={"ID":"132ce48a-6190-4f71-8b77-1a28c917030d","Type":"ContainerDied","Data":"af71e4ba7de3011808a6e0423695dc00d10d84eaa2810dcf4a330acc8dc2e164"} Oct 30 22:32:18.175224 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:18.175019 6720 scope.go:122] "RemoveContainer" containerID="e23c19265f41345552b83a4b4160b4d096b58b4e126e3777ca42e8f6497dbfee" Oct 30 22:32:18.199527 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:18.199499 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-bacea0-58688dc49f-m4cv7"] Oct 30 22:32:18.203625 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:18.203597 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-bacea0-58688dc49f-m4cv7"] Oct 30 22:32:18.763407 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:18.761946 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:19.560934 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:19.560887 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:19.957667 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:19.957630 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="132ce48a-6190-4f71-8b77-1a28c917030d" path="/var/lib/kubelet/pods/132ce48a-6190-4f71-8b77-1a28c917030d/volumes" Oct 30 22:32:20.257062 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:20.256972 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:20.443772 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:20.443732 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:20.591121 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:20.591035 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:20.778754 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:20.778720 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:21.130816 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:21.130771 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:22.028409 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:22.028344 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:22.836966 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:22.836916 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:23.524414 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:23.518005 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:23.854617 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:23.854570 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:24.008169 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:24.008134 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:24.161710 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:24.161619 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:24.324123 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:24.324092 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:24.954152 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:24.954070 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:25.638191 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:25.638131 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:25.812050 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:25.812017 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:25.975776 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:25.975665 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:26.680573 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:26.680540 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:27.313492 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:27.313456 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:28.011736 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:28.011701 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:28.646843 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:28.646758 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:29.449661 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:29.449627 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:30.127161 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:30.127123 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:30.303516 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:30.303485 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:30.823913 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:30.823854 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:31.439191 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:31.439161 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:31.590562 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:31.590529 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:32.006955 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:32.006922 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:32.241154 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:32.241120 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:32.404944 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:32.404909 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:32.813260 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:32.813169 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:33.274176 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:33.274115 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:33.955256 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:33.955216 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:34.646231 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:34.646190 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:34.880148 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:34.880114 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:35.018861 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:35.018777 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:35.212268 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:35.212236 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:35.397421 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:35.395264 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:35.552328 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:35.552297 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:35.877679 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:35.877629 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:36.427451 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:36.425778 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:37.218543 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:37.218505 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:38.008261 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:38.008230 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:38.845201 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:38.845161 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:39.410297 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:39.410257 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:39.836070 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:39.835981 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-a1ea1e-5b7bb88c4b-f6w5x"] Oct 30 22:32:39.836290 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:39.836249 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a1ea1e-5b7bb88c4b-f6w5x" podUID="083d25b4-5991-4298-b2f3-392d28c4e620" containerName="maas-api" containerID="cri-o://b4e09fe63d07362925fd068e98d052f6e5bd5fdd45aa84498da411a31fde3ad4" gracePeriod=30 Oct 30 22:32:39.923060 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:39.923026 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:40.072681 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:40.069639 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:40.148196 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:40.148160 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a1ea1e-5b7bb88c4b-f6w5x" Oct 30 22:32:40.219860 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:40.219783 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:40.353755 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:40.353726 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/083d25b4-5991-4298-b2f3-392d28c4e620-metrics-certs\" (UniqueName: \"kubernetes.io/secret/083d25b4-5991-4298-b2f3-392d28c4e620-metrics-certs\") pod \"083d25b4-5991-4298-b2f3-392d28c4e620\" (UID: \"083d25b4-5991-4298-b2f3-392d28c4e620\") " Oct 30 22:32:40.353957 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:40.353830 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/083d25b4-5991-4298-b2f3-392d28c4e620-kube-api-access-svmmm\" (UniqueName: \"kubernetes.io/projected/083d25b4-5991-4298-b2f3-392d28c4e620-kube-api-access-svmmm\") pod \"083d25b4-5991-4298-b2f3-392d28c4e620\" (UID: \"083d25b4-5991-4298-b2f3-392d28c4e620\") " Oct 30 22:32:40.353957 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:40.353882 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/083d25b4-5991-4298-b2f3-392d28c4e620-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/083d25b4-5991-4298-b2f3-392d28c4e620-maas-api-tls\") pod \"083d25b4-5991-4298-b2f3-392d28c4e620\" (UID: \"083d25b4-5991-4298-b2f3-392d28c4e620\") " Oct 30 22:32:40.354076 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:40.354032 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/083d25b4-5991-4298-b2f3-392d28c4e620-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/083d25b4-5991-4298-b2f3-392d28c4e620-openshift-service-ca\") pod \"083d25b4-5991-4298-b2f3-392d28c4e620\" (UID: \"083d25b4-5991-4298-b2f3-392d28c4e620\") " Oct 30 22:32:40.354579 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:40.354550 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/083d25b4-5991-4298-b2f3-392d28c4e620-openshift-service-ca" pod "083d25b4-5991-4298-b2f3-392d28c4e620" (UID: "083d25b4-5991-4298-b2f3-392d28c4e620"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:32:40.357478 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:40.357453 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/083d25b4-5991-4298-b2f3-392d28c4e620-metrics-certs" pod "083d25b4-5991-4298-b2f3-392d28c4e620" (UID: "083d25b4-5991-4298-b2f3-392d28c4e620"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:32:40.358879 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:40.358849 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/083d25b4-5991-4298-b2f3-392d28c4e620-kube-api-access-svmmm" pod "083d25b4-5991-4298-b2f3-392d28c4e620" (UID: "083d25b4-5991-4298-b2f3-392d28c4e620"). InnerVolumeSpecName "kube-api-access-svmmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:32:40.358879 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:40.358868 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/083d25b4-5991-4298-b2f3-392d28c4e620-maas-api-tls" pod "083d25b4-5991-4298-b2f3-392d28c4e620" (UID: "083d25b4-5991-4298-b2f3-392d28c4e620"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:32:40.368197 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:40.368172 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:40.459714 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:32:40.459016 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 22:32:40.459714 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:40.459369 6720 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/083d25b4-5991-4298-b2f3-392d28c4e620-openshift-service-ca\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:40.459714 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:40.459413 6720 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/083d25b4-5991-4298-b2f3-392d28c4e620-metrics-certs\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:40.459714 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:40.459428 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svmmm\" (UniqueName: \"kubernetes.io/projected/083d25b4-5991-4298-b2f3-392d28c4e620-kube-api-access-svmmm\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:40.459714 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:40.459441 6720 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/083d25b4-5991-4298-b2f3-392d28c4e620-maas-api-tls\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:32:40.467559 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:40.463782 6720 generic.go:356] "Generic (PLEG): container finished" podID="083d25b4-5991-4298-b2f3-392d28c4e620" containerID="b4e09fe63d07362925fd068e98d052f6e5bd5fdd45aa84498da411a31fde3ad4" exitCode=0 Oct 30 22:32:40.467559 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:40.463976 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a1ea1e-5b7bb88c4b-f6w5x" event={"ID":"083d25b4-5991-4298-b2f3-392d28c4e620","Type":"ContainerDied","Data":"b4e09fe63d07362925fd068e98d052f6e5bd5fdd45aa84498da411a31fde3ad4"} Oct 30 22:32:40.467559 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:40.464026 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a1ea1e-5b7bb88c4b-f6w5x" event={"ID":"083d25b4-5991-4298-b2f3-392d28c4e620","Type":"ContainerDied","Data":"d5a81d720abb8d2ee9dcd87cb196612cd2a11e5ff07de0cfa107dd7dd605c398"} Oct 30 22:32:40.467559 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:40.464051 6720 scope.go:122] "RemoveContainer" containerID="b4e09fe63d07362925fd068e98d052f6e5bd5fdd45aa84498da411a31fde3ad4" Oct 30 22:32:40.467559 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:40.464223 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a1ea1e-5b7bb88c4b-f6w5x" Oct 30 22:32:40.516409 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:40.516014 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-a1ea1e-5b7bb88c4b-f6w5x"] Oct 30 22:32:40.516409 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:40.516105 6720 scope.go:122] "RemoveContainer" containerID="b4e09fe63d07362925fd068e98d052f6e5bd5fdd45aa84498da411a31fde3ad4" Oct 30 22:32:40.516601 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:32:40.516481 6720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4e09fe63d07362925fd068e98d052f6e5bd5fdd45aa84498da411a31fde3ad4\": container with ID starting with b4e09fe63d07362925fd068e98d052f6e5bd5fdd45aa84498da411a31fde3ad4 not found: ID does not exist" containerID="b4e09fe63d07362925fd068e98d052f6e5bd5fdd45aa84498da411a31fde3ad4" Oct 30 22:32:40.516601 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:40.516514 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4e09fe63d07362925fd068e98d052f6e5bd5fdd45aa84498da411a31fde3ad4"} err="failed to get container status \"b4e09fe63d07362925fd068e98d052f6e5bd5fdd45aa84498da411a31fde3ad4\": rpc error: code = NotFound desc = could not find container \"b4e09fe63d07362925fd068e98d052f6e5bd5fdd45aa84498da411a31fde3ad4\": container with ID starting with b4e09fe63d07362925fd068e98d052f6e5bd5fdd45aa84498da411a31fde3ad4 not found: ID does not exist" Oct 30 22:32:40.517984 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:40.517966 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-a1ea1e-5b7bb88c4b-f6w5x"] Oct 30 22:32:40.711153 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:40.711068 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:41.417280 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:41.413349 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:41.957319 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:41.957286 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="083d25b4-5991-4298-b2f3-392d28c4e620" path="/var/lib/kubelet/pods/083d25b4-5991-4298-b2f3-392d28c4e620/volumes" Oct 30 22:32:42.043023 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:42.042987 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:42.730359 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:42.730323 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:43.162209 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:43.162170 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:43.315253 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:43.315217 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:43.505121 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:43.505030 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:43.656722 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:43.656689 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:43.914469 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:43.914427 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:44.346883 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:44.346786 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:45.174798 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:45.174750 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:45.831561 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:45.824990 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:46.539837 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:46.539797 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:47.105017 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:47.104975 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:47.287348 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:47.287310 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:47.468931 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:47.468847 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:47.616793 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:47.616765 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:48.132862 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:48.132811 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:48.856644 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:48.856605 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:48.869087 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:48.869045 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-0f26c5/distinct-2-w1-0f26c5-kserve-74976f985c-fw28x"] Oct 30 22:32:48.876884 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:48.876845 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-0f26c5/distinct-w1-0f26c5-kserve-7d6b79498b-lvsjw"] Oct 30 22:32:49.038149 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:49.038117 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:49.590019 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:49.589985 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:50.209134 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:50.209096 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:50.919700 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:50.919665 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:51.107477 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:51.107443 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:51.289305 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:51.289219 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:51.702143 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:51.702105 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:52.522749 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:52.522692 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:53.150909 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:53.150858 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:53.748071 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:53.748030 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:54.361105 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:54.361072 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:54.519854 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:54.519823 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:54.686182 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:54.686082 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:55.223514 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:55.223481 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:56.035155 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:56.035112 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:56.743908 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:56.743872 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:57.520081 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:57.520048 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:58.219798 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:58.219766 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:58.375782 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:58.375747 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:58.927463 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:58.927424 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:32:59.524208 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:32:59.524173 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:00.258025 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:00.254320 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:00.894119 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:00.894081 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:01.481969 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:01.481933 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-a5f192-585685f64d-2vwwn"] Oct 30 22:33:01.483100 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:01.483066 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a5f192-585685f64d-2vwwn" podUID="423a4a57-1e84-4833-ac24-261dcad52016" containerName="maas-api" containerID="cri-o://e903569b3b6be233f8af498e382f7d34441979c029700e52fb4e5e76ecd71833" gracePeriod=30 Oct 30 22:33:01.579394 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:01.579362 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:01.760515 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:01.743632 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:01.790104 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:01.761021 6720 generic.go:356] "Generic (PLEG): container finished" podID="423a4a57-1e84-4833-ac24-261dcad52016" containerID="e903569b3b6be233f8af498e382f7d34441979c029700e52fb4e5e76ecd71833" exitCode=0 Oct 30 22:33:01.790104 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:01.761221 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a5f192-585685f64d-2vwwn" event={"ID":"423a4a57-1e84-4833-ac24-261dcad52016","Type":"ContainerDied","Data":"e903569b3b6be233f8af498e382f7d34441979c029700e52fb4e5e76ecd71833"} Oct 30 22:33:01.891930 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:01.891614 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:01.897941 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:01.897912 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a5f192-585685f64d-2vwwn" Oct 30 22:33:01.991305 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:01.991278 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/423a4a57-1e84-4833-ac24-261dcad52016-kube-api-access-dbxnq\" (UniqueName: \"kubernetes.io/projected/423a4a57-1e84-4833-ac24-261dcad52016-kube-api-access-dbxnq\") pod \"423a4a57-1e84-4833-ac24-261dcad52016\" (UID: \"423a4a57-1e84-4833-ac24-261dcad52016\") " Oct 30 22:33:01.991546 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:01.991319 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/423a4a57-1e84-4833-ac24-261dcad52016-metrics-certs\" (UniqueName: \"kubernetes.io/secret/423a4a57-1e84-4833-ac24-261dcad52016-metrics-certs\") pod \"423a4a57-1e84-4833-ac24-261dcad52016\" (UID: \"423a4a57-1e84-4833-ac24-261dcad52016\") " Oct 30 22:33:01.991546 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:01.991461 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/423a4a57-1e84-4833-ac24-261dcad52016-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/423a4a57-1e84-4833-ac24-261dcad52016-openshift-service-ca\") pod \"423a4a57-1e84-4833-ac24-261dcad52016\" (UID: \"423a4a57-1e84-4833-ac24-261dcad52016\") " Oct 30 22:33:01.991546 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:01.991507 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/423a4a57-1e84-4833-ac24-261dcad52016-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/423a4a57-1e84-4833-ac24-261dcad52016-maas-api-tls\") pod \"423a4a57-1e84-4833-ac24-261dcad52016\" (UID: \"423a4a57-1e84-4833-ac24-261dcad52016\") " Oct 30 22:33:01.991770 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:01.991741 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/423a4a57-1e84-4833-ac24-261dcad52016-openshift-service-ca" pod "423a4a57-1e84-4833-ac24-261dcad52016" (UID: "423a4a57-1e84-4833-ac24-261dcad52016"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:33:01.992087 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:01.992059 6720 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/423a4a57-1e84-4833-ac24-261dcad52016-openshift-service-ca\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:01.993905 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:01.993875 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/423a4a57-1e84-4833-ac24-261dcad52016-maas-api-tls" pod "423a4a57-1e84-4833-ac24-261dcad52016" (UID: "423a4a57-1e84-4833-ac24-261dcad52016"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:33:01.994010 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:01.993933 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/423a4a57-1e84-4833-ac24-261dcad52016-metrics-certs" pod "423a4a57-1e84-4833-ac24-261dcad52016" (UID: "423a4a57-1e84-4833-ac24-261dcad52016"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:33:01.994078 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:01.994028 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/423a4a57-1e84-4833-ac24-261dcad52016-kube-api-access-dbxnq" pod "423a4a57-1e84-4833-ac24-261dcad52016" (UID: "423a4a57-1e84-4833-ac24-261dcad52016"). InnerVolumeSpecName "kube-api-access-dbxnq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:33:02.042628 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:02.042523 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:02.093188 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:02.093153 6720 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/423a4a57-1e84-4833-ac24-261dcad52016-maas-api-tls\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:02.093188 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:02.093191 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbxnq\" (UniqueName: \"kubernetes.io/projected/423a4a57-1e84-4833-ac24-261dcad52016-kube-api-access-dbxnq\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:02.093513 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:02.093210 6720 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/423a4a57-1e84-4833-ac24-261dcad52016-metrics-certs\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:02.429015 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:02.428977 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:02.768033 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:02.767951 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a5f192-585685f64d-2vwwn" Oct 30 22:33:02.768033 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:02.767987 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-a5f192-585685f64d-2vwwn" event={"ID":"423a4a57-1e84-4833-ac24-261dcad52016","Type":"ContainerDied","Data":"fbfb890fa87340811035202c92760e4da9c2e8b473933ee509f28e07682e3739"} Oct 30 22:33:02.768243 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:02.768054 6720 scope.go:122] "RemoveContainer" containerID="e903569b3b6be233f8af498e382f7d34441979c029700e52fb4e5e76ecd71833" Oct 30 22:33:02.795810 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:02.795779 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-a5f192-585685f64d-2vwwn"] Oct 30 22:33:02.801777 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:02.801750 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-a5f192-585685f64d-2vwwn"] Oct 30 22:33:02.924670 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:02.924479 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:03.600418 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:03.600358 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:03.905214 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:03.905169 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-0f26c5/distinct-w1-0f26c5-kserve-7d6b79498b-lvsjw" podUID="457d2f97-63ab-41f4-b761-294d0ba3045e" containerName="main" containerID="cri-o://e3beec726f3388452ebce7cdd62b42b8a234745f8d7cdf1dc83f6216d7dcf25a" gracePeriod=45 Oct 30 22:33:03.905432 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:03.905331 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-0f26c5/distinct-2-w1-0f26c5-kserve-74976f985c-fw28x" podUID="97e1e65d-1d50-4097-98ca-2944fa182807" containerName="main" containerID="cri-o://96f2583dfd9b4c597466bcd32b75582e9764458cc65ff98e4dc8c30aff0e8176" gracePeriod=45 Oct 30 22:33:03.957630 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:03.957605 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="423a4a57-1e84-4833-ac24-261dcad52016" path="/var/lib/kubelet/pods/423a4a57-1e84-4833-ac24-261dcad52016/volumes" Oct 30 22:33:04.128426 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:04.125664 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:04.383626 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:04.354608 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:04.408363 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:04.408288 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-0f26c5/distinct-2-w1-0f26c5-kserve-74976f985c-fw28x" Oct 30 22:33:04.470547 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:04.470126 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-0f26c5/distinct-w1-0f26c5-kserve-7d6b79498b-lvsjw" Oct 30 22:33:04.507787 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:04.507749 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/457d2f97-63ab-41f4-b761-294d0ba3045e-home\" (UniqueName: \"kubernetes.io/empty-dir/457d2f97-63ab-41f4-b761-294d0ba3045e-home\") pod \"457d2f97-63ab-41f4-b761-294d0ba3045e\" (UID: \"457d2f97-63ab-41f4-b761-294d0ba3045e\") " Oct 30 22:33:04.507978 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:04.507801 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97e1e65d-1d50-4097-98ca-2944fa182807-kube-api-access-k88td\" (UniqueName: \"kubernetes.io/projected/97e1e65d-1d50-4097-98ca-2944fa182807-kube-api-access-k88td\") pod \"97e1e65d-1d50-4097-98ca-2944fa182807\" (UID: \"97e1e65d-1d50-4097-98ca-2944fa182807\") " Oct 30 22:33:04.507978 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:04.507855 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/457d2f97-63ab-41f4-b761-294d0ba3045e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/457d2f97-63ab-41f4-b761-294d0ba3045e-model-cache\") pod \"457d2f97-63ab-41f4-b761-294d0ba3045e\" (UID: \"457d2f97-63ab-41f4-b761-294d0ba3045e\") " Oct 30 22:33:04.507978 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:04.507914 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/457d2f97-63ab-41f4-b761-294d0ba3045e-kube-api-access-4qckh\" (UniqueName: \"kubernetes.io/projected/457d2f97-63ab-41f4-b761-294d0ba3045e-kube-api-access-4qckh\") pod \"457d2f97-63ab-41f4-b761-294d0ba3045e\" (UID: \"457d2f97-63ab-41f4-b761-294d0ba3045e\") " Oct 30 22:33:04.507978 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:04.507953 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97e1e65d-1d50-4097-98ca-2944fa182807-home\" (UniqueName: \"kubernetes.io/empty-dir/97e1e65d-1d50-4097-98ca-2944fa182807-home\") pod \"97e1e65d-1d50-4097-98ca-2944fa182807\" (UID: \"97e1e65d-1d50-4097-98ca-2944fa182807\") " Oct 30 22:33:04.508250 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:04.508006 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97e1e65d-1d50-4097-98ca-2944fa182807-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97e1e65d-1d50-4097-98ca-2944fa182807-model-cache\") pod \"97e1e65d-1d50-4097-98ca-2944fa182807\" (UID: \"97e1e65d-1d50-4097-98ca-2944fa182807\") " Oct 30 22:33:04.508250 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:04.508047 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97e1e65d-1d50-4097-98ca-2944fa182807-dshm\" (UniqueName: \"kubernetes.io/empty-dir/97e1e65d-1d50-4097-98ca-2944fa182807-dshm\") pod \"97e1e65d-1d50-4097-98ca-2944fa182807\" (UID: \"97e1e65d-1d50-4097-98ca-2944fa182807\") " Oct 30 22:33:04.508250 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:04.508086 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/457d2f97-63ab-41f4-b761-294d0ba3045e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/457d2f97-63ab-41f4-b761-294d0ba3045e-dshm\") pod \"457d2f97-63ab-41f4-b761-294d0ba3045e\" (UID: \"457d2f97-63ab-41f4-b761-294d0ba3045e\") " Oct 30 22:33:04.508250 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:04.508149 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97e1e65d-1d50-4097-98ca-2944fa182807-tls-certs\" (UniqueName: \"kubernetes.io/secret/97e1e65d-1d50-4097-98ca-2944fa182807-tls-certs\") pod \"97e1e65d-1d50-4097-98ca-2944fa182807\" (UID: \"97e1e65d-1d50-4097-98ca-2944fa182807\") " Oct 30 22:33:04.508521 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:04.508258 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/457d2f97-63ab-41f4-b761-294d0ba3045e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/457d2f97-63ab-41f4-b761-294d0ba3045e-tmp-dir\") pod \"457d2f97-63ab-41f4-b761-294d0ba3045e\" (UID: \"457d2f97-63ab-41f4-b761-294d0ba3045e\") " Oct 30 22:33:04.508521 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:04.508294 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/457d2f97-63ab-41f4-b761-294d0ba3045e-tls-certs\" (UniqueName: \"kubernetes.io/secret/457d2f97-63ab-41f4-b761-294d0ba3045e-tls-certs\") pod \"457d2f97-63ab-41f4-b761-294d0ba3045e\" (UID: \"457d2f97-63ab-41f4-b761-294d0ba3045e\") " Oct 30 22:33:04.508521 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:04.508365 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97e1e65d-1d50-4097-98ca-2944fa182807-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97e1e65d-1d50-4097-98ca-2944fa182807-tmp-dir\") pod \"97e1e65d-1d50-4097-98ca-2944fa182807\" (UID: \"97e1e65d-1d50-4097-98ca-2944fa182807\") " Oct 30 22:33:04.509307 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:04.509279 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97e1e65d-1d50-4097-98ca-2944fa182807-model-cache" pod "97e1e65d-1d50-4097-98ca-2944fa182807" (UID: "97e1e65d-1d50-4097-98ca-2944fa182807"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:33:04.509528 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:04.509506 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/457d2f97-63ab-41f4-b761-294d0ba3045e-home" pod "457d2f97-63ab-41f4-b761-294d0ba3045e" (UID: "457d2f97-63ab-41f4-b761-294d0ba3045e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:33:04.509829 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:04.509809 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97e1e65d-1d50-4097-98ca-2944fa182807-tmp-dir" pod "97e1e65d-1d50-4097-98ca-2944fa182807" (UID: "97e1e65d-1d50-4097-98ca-2944fa182807"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:33:04.510919 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:04.510891 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/457d2f97-63ab-41f4-b761-294d0ba3045e-model-cache" pod "457d2f97-63ab-41f4-b761-294d0ba3045e" (UID: "457d2f97-63ab-41f4-b761-294d0ba3045e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:33:04.511364 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:04.511336 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97e1e65d-1d50-4097-98ca-2944fa182807-home" pod "97e1e65d-1d50-4097-98ca-2944fa182807" (UID: "97e1e65d-1d50-4097-98ca-2944fa182807"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:33:04.512512 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:04.512460 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/457d2f97-63ab-41f4-b761-294d0ba3045e-tmp-dir" pod "457d2f97-63ab-41f4-b761-294d0ba3045e" (UID: "457d2f97-63ab-41f4-b761-294d0ba3045e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:33:04.517433 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:04.517367 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/457d2f97-63ab-41f4-b761-294d0ba3045e-dshm" pod "457d2f97-63ab-41f4-b761-294d0ba3045e" (UID: "457d2f97-63ab-41f4-b761-294d0ba3045e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:33:04.517541 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:04.517467 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/457d2f97-63ab-41f4-b761-294d0ba3045e-kube-api-access-4qckh" pod "457d2f97-63ab-41f4-b761-294d0ba3045e" (UID: "457d2f97-63ab-41f4-b761-294d0ba3045e"). InnerVolumeSpecName "kube-api-access-4qckh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:33:04.517541 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:04.517478 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97e1e65d-1d50-4097-98ca-2944fa182807-kube-api-access-k88td" pod "97e1e65d-1d50-4097-98ca-2944fa182807" (UID: "97e1e65d-1d50-4097-98ca-2944fa182807"). InnerVolumeSpecName "kube-api-access-k88td". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:33:04.517727 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:04.517572 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97e1e65d-1d50-4097-98ca-2944fa182807-dshm" pod "97e1e65d-1d50-4097-98ca-2944fa182807" (UID: "97e1e65d-1d50-4097-98ca-2944fa182807"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:33:04.518813 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:04.518787 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97e1e65d-1d50-4097-98ca-2944fa182807-tls-certs" pod "97e1e65d-1d50-4097-98ca-2944fa182807" (UID: "97e1e65d-1d50-4097-98ca-2944fa182807"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:33:04.519121 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:04.519028 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/457d2f97-63ab-41f4-b761-294d0ba3045e-tls-certs" pod "457d2f97-63ab-41f4-b761-294d0ba3045e" (UID: "457d2f97-63ab-41f4-b761-294d0ba3045e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:33:04.522824 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:04.522796 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:04.610899 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:04.610684 6720 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97e1e65d-1d50-4097-98ca-2944fa182807-tmp-dir\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:04.610899 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:04.610726 6720 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/457d2f97-63ab-41f4-b761-294d0ba3045e-home\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:04.610899 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:04.610751 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k88td\" (UniqueName: \"kubernetes.io/projected/97e1e65d-1d50-4097-98ca-2944fa182807-kube-api-access-k88td\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:04.610899 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:04.610768 6720 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/457d2f97-63ab-41f4-b761-294d0ba3045e-model-cache\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:04.610899 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:04.610784 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qckh\" (UniqueName: \"kubernetes.io/projected/457d2f97-63ab-41f4-b761-294d0ba3045e-kube-api-access-4qckh\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:04.610899 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:04.610798 6720 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97e1e65d-1d50-4097-98ca-2944fa182807-home\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:04.610899 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:04.610812 6720 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97e1e65d-1d50-4097-98ca-2944fa182807-model-cache\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:04.610899 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:04.610824 6720 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97e1e65d-1d50-4097-98ca-2944fa182807-dshm\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:04.610899 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:04.610836 6720 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/457d2f97-63ab-41f4-b761-294d0ba3045e-dshm\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:04.610899 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:04.610848 6720 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97e1e65d-1d50-4097-98ca-2944fa182807-tls-certs\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:04.610899 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:04.610859 6720 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/457d2f97-63ab-41f4-b761-294d0ba3045e-tmp-dir\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:04.610899 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:04.610871 6720 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/457d2f97-63ab-41f4-b761-294d0ba3045e-tls-certs\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:04.676372 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:04.676314 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:04.783083 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:04.782990 6720 generic.go:356] "Generic (PLEG): container finished" podID="457d2f97-63ab-41f4-b761-294d0ba3045e" containerID="e3beec726f3388452ebce7cdd62b42b8a234745f8d7cdf1dc83f6216d7dcf25a" exitCode=0 Oct 30 22:33:04.783083 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:04.783049 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-0f26c5/distinct-w1-0f26c5-kserve-7d6b79498b-lvsjw" Oct 30 22:33:04.783330 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:04.783081 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-0f26c5/distinct-w1-0f26c5-kserve-7d6b79498b-lvsjw" event={"ID":"457d2f97-63ab-41f4-b761-294d0ba3045e","Type":"ContainerDied","Data":"e3beec726f3388452ebce7cdd62b42b8a234745f8d7cdf1dc83f6216d7dcf25a"} Oct 30 22:33:04.783330 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:04.783134 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-0f26c5/distinct-w1-0f26c5-kserve-7d6b79498b-lvsjw" event={"ID":"457d2f97-63ab-41f4-b761-294d0ba3045e","Type":"ContainerDied","Data":"1fb3f2dd4f80080bd6e042321b4bd014a921f060a2adac73a3edbc7cd51e292f"} Oct 30 22:33:04.783330 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:04.783159 6720 scope.go:122] "RemoveContainer" containerID="e3beec726f3388452ebce7cdd62b42b8a234745f8d7cdf1dc83f6216d7dcf25a" Oct 30 22:33:04.784827 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:04.784797 6720 generic.go:356] "Generic (PLEG): container finished" podID="97e1e65d-1d50-4097-98ca-2944fa182807" containerID="96f2583dfd9b4c597466bcd32b75582e9764458cc65ff98e4dc8c30aff0e8176" exitCode=0 Oct 30 22:33:04.784980 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:04.784832 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-0f26c5/distinct-2-w1-0f26c5-kserve-74976f985c-fw28x" event={"ID":"97e1e65d-1d50-4097-98ca-2944fa182807","Type":"ContainerDied","Data":"96f2583dfd9b4c597466bcd32b75582e9764458cc65ff98e4dc8c30aff0e8176"} Oct 30 22:33:04.784980 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:04.784861 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-0f26c5/distinct-2-w1-0f26c5-kserve-74976f985c-fw28x" Oct 30 22:33:04.785223 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:04.784862 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-0f26c5/distinct-2-w1-0f26c5-kserve-74976f985c-fw28x" event={"ID":"97e1e65d-1d50-4097-98ca-2944fa182807","Type":"ContainerDied","Data":"fdc4418bc41b83d626bab237150b3c67dde8cc6e0fa5feefbfb785e1ff9b5f30"} Oct 30 22:33:04.799847 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:04.799277 6720 scope.go:122] "RemoveContainer" containerID="e3beec726f3388452ebce7cdd62b42b8a234745f8d7cdf1dc83f6216d7dcf25a" Oct 30 22:33:04.799847 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:33:04.799691 6720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3beec726f3388452ebce7cdd62b42b8a234745f8d7cdf1dc83f6216d7dcf25a\": container with ID starting with e3beec726f3388452ebce7cdd62b42b8a234745f8d7cdf1dc83f6216d7dcf25a not found: ID does not exist" containerID="e3beec726f3388452ebce7cdd62b42b8a234745f8d7cdf1dc83f6216d7dcf25a" Oct 30 22:33:04.799847 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:04.799730 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3beec726f3388452ebce7cdd62b42b8a234745f8d7cdf1dc83f6216d7dcf25a"} err="failed to get container status \"e3beec726f3388452ebce7cdd62b42b8a234745f8d7cdf1dc83f6216d7dcf25a\": rpc error: code = NotFound desc = could not find container \"e3beec726f3388452ebce7cdd62b42b8a234745f8d7cdf1dc83f6216d7dcf25a\": container with ID starting with e3beec726f3388452ebce7cdd62b42b8a234745f8d7cdf1dc83f6216d7dcf25a not found: ID does not exist" Oct 30 22:33:04.799847 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:04.799752 6720 scope.go:122] "RemoveContainer" containerID="96f2583dfd9b4c597466bcd32b75582e9764458cc65ff98e4dc8c30aff0e8176" Oct 30 22:33:04.810537 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:04.810512 6720 scope.go:122] "RemoveContainer" containerID="96f2583dfd9b4c597466bcd32b75582e9764458cc65ff98e4dc8c30aff0e8176" Oct 30 22:33:04.810837 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:33:04.810811 6720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96f2583dfd9b4c597466bcd32b75582e9764458cc65ff98e4dc8c30aff0e8176\": container with ID starting with 96f2583dfd9b4c597466bcd32b75582e9764458cc65ff98e4dc8c30aff0e8176 not found: ID does not exist" containerID="96f2583dfd9b4c597466bcd32b75582e9764458cc65ff98e4dc8c30aff0e8176" Oct 30 22:33:04.810946 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:04.810845 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96f2583dfd9b4c597466bcd32b75582e9764458cc65ff98e4dc8c30aff0e8176"} err="failed to get container status \"96f2583dfd9b4c597466bcd32b75582e9764458cc65ff98e4dc8c30aff0e8176\": rpc error: code = NotFound desc = could not find container \"96f2583dfd9b4c597466bcd32b75582e9764458cc65ff98e4dc8c30aff0e8176\": container with ID starting with 96f2583dfd9b4c597466bcd32b75582e9764458cc65ff98e4dc8c30aff0e8176 not found: ID does not exist" Oct 30 22:33:04.820613 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:04.820591 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-0f26c5/distinct-w1-0f26c5-kserve-7d6b79498b-lvsjw"] Oct 30 22:33:04.824323 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:04.824302 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-0f26c5/distinct-w1-0f26c5-kserve-7d6b79498b-lvsjw"] Oct 30 22:33:04.838447 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:04.838423 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-0f26c5/distinct-2-w1-0f26c5-kserve-74976f985c-fw28x"] Oct 30 22:33:04.846565 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:04.846542 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-0f26c5/distinct-2-w1-0f26c5-kserve-74976f985c-fw28x"] Oct 30 22:33:05.032541 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:05.032500 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:05.332433 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:05.332359 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-0f26c5/distinct-2-w1-0f26c5-kserve-74976f985c-fw28x" podUID="97e1e65d-1d50-4097-98ca-2944fa182807" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:33:05.424938 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:05.423051 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:05.956873 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:05.956841 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="457d2f97-63ab-41f4-b761-294d0ba3045e" path="/var/lib/kubelet/pods/457d2f97-63ab-41f4-b761-294d0ba3045e/volumes" Oct 30 22:33:05.957289 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:05.957276 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97e1e65d-1d50-4097-98ca-2944fa182807" path="/var/lib/kubelet/pods/97e1e65d-1d50-4097-98ca-2944fa182807/volumes" Oct 30 22:33:06.139414 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:06.138858 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:06.730076 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:06.729566 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:07.001531 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:07.001439 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:07.139687 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:07.139655 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:07.425015 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:07.423890 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:08.048833 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:08.048797 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:08.880542 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:08.880505 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:09.363460 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:09.363361 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:09.627200 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:09.627114 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:09.793786 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:09.793750 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:10.410296 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:33:10.410257 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 22:33:10.527136 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:10.526980 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:12.973973 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:12.973943 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:13.166917 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:13.166881 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:13.354362 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:13.354334 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:14.274608 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:14.274469 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:14.990131 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:14.990095 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:15.744588 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:15.744556 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:16.678125 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:16.678086 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:17.321206 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:17.321136 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:18.034328 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:18.034283 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:18.477178 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:18.477140 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:18.696039 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:18.696006 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:18.839842 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:18.839754 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:19.404987 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:19.404951 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:20.178653 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:20.178620 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:20.474611 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:20.474528 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:20.637516 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:20.637480 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:21.120114 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:21.120071 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:21.886937 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:21.886901 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:22.532992 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:22.532939 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:22.845262 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:22.845162 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:23.506231 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:23.506176 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:23.901223 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:23.901187 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:24.048454 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:24.048419 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:24.426276 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:24.426191 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:25.222413 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:25.218943 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:26.054655 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:26.054620 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:26.594110 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:26.594072 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:26.751235 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:26.751189 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:26.912295 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:26.912260 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:27.395570 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:27.395534 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:27.537945 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:27.537907 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:28.119886 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:28.119832 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:28.902187 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:28.902147 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:29.566886 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:29.566845 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:30.134929 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:30.134895 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:30.387173 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:30.387090 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:30.559117 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:30.559084 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:30.912676 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:30.912636 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:31.021819 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:31.019997 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0a2577-7f97948b44-d669v"] Oct 30 22:33:31.021819 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:31.020248 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0a2577-7f97948b44-d669v" podUID="7365fe23-d73c-467d-bd8b-d0177976452f" containerName="maas-api" containerID="cri-o://6aabc6ce1727c178c59ff358db85a05f4fb1bd5e7e7f669143faad61ccddee65" gracePeriod=30 Oct 30 22:33:31.113705 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:31.113657 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:31.321659 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:31.289256 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:31.447357 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:31.447325 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:31.473602 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:31.473578 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0a2577-7f97948b44-d669v" Oct 30 22:33:31.523959 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:31.523863 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7365fe23-d73c-467d-bd8b-d0177976452f-kube-api-access-bk56p\" (UniqueName: \"kubernetes.io/projected/7365fe23-d73c-467d-bd8b-d0177976452f-kube-api-access-bk56p\") pod \"7365fe23-d73c-467d-bd8b-d0177976452f\" (UID: \"7365fe23-d73c-467d-bd8b-d0177976452f\") " Oct 30 22:33:31.524136 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:31.523999 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7365fe23-d73c-467d-bd8b-d0177976452f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7365fe23-d73c-467d-bd8b-d0177976452f-metrics-certs\") pod \"7365fe23-d73c-467d-bd8b-d0177976452f\" (UID: \"7365fe23-d73c-467d-bd8b-d0177976452f\") " Oct 30 22:33:31.524136 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:31.524072 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7365fe23-d73c-467d-bd8b-d0177976452f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7365fe23-d73c-467d-bd8b-d0177976452f-maas-api-tls\") pod \"7365fe23-d73c-467d-bd8b-d0177976452f\" (UID: \"7365fe23-d73c-467d-bd8b-d0177976452f\") " Oct 30 22:33:31.524270 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:31.524184 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7365fe23-d73c-467d-bd8b-d0177976452f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7365fe23-d73c-467d-bd8b-d0177976452f-openshift-service-ca\") pod \"7365fe23-d73c-467d-bd8b-d0177976452f\" (UID: \"7365fe23-d73c-467d-bd8b-d0177976452f\") " Oct 30 22:33:31.524876 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:31.524841 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7365fe23-d73c-467d-bd8b-d0177976452f-openshift-service-ca" pod "7365fe23-d73c-467d-bd8b-d0177976452f" (UID: "7365fe23-d73c-467d-bd8b-d0177976452f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:33:31.526328 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:31.526300 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7365fe23-d73c-467d-bd8b-d0177976452f-kube-api-access-bk56p" pod "7365fe23-d73c-467d-bd8b-d0177976452f" (UID: "7365fe23-d73c-467d-bd8b-d0177976452f"). InnerVolumeSpecName "kube-api-access-bk56p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:33:31.526961 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:31.526927 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7365fe23-d73c-467d-bd8b-d0177976452f-maas-api-tls" pod "7365fe23-d73c-467d-bd8b-d0177976452f" (UID: "7365fe23-d73c-467d-bd8b-d0177976452f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:33:31.527084 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:31.526996 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7365fe23-d73c-467d-bd8b-d0177976452f-metrics-certs" pod "7365fe23-d73c-467d-bd8b-d0177976452f" (UID: "7365fe23-d73c-467d-bd8b-d0177976452f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:33:31.590455 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:31.590372 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:31.626477 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:31.626434 6720 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7365fe23-d73c-467d-bd8b-d0177976452f-openshift-service-ca\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:31.626903 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:31.626876 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bk56p\" (UniqueName: \"kubernetes.io/projected/7365fe23-d73c-467d-bd8b-d0177976452f-kube-api-access-bk56p\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:31.627035 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:31.627022 6720 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7365fe23-d73c-467d-bd8b-d0177976452f-metrics-certs\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:31.627119 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:31.627108 6720 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7365fe23-d73c-467d-bd8b-d0177976452f-maas-api-tls\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:31.932027 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:31.931995 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:32.020456 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:32.020422 6720 generic.go:356] "Generic (PLEG): container finished" podID="7365fe23-d73c-467d-bd8b-d0177976452f" containerID="6aabc6ce1727c178c59ff358db85a05f4fb1bd5e7e7f669143faad61ccddee65" exitCode=0 Oct 30 22:33:32.020618 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:32.020477 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0a2577-7f97948b44-d669v" Oct 30 22:33:32.020618 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:32.020509 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0a2577-7f97948b44-d669v" event={"ID":"7365fe23-d73c-467d-bd8b-d0177976452f","Type":"ContainerDied","Data":"6aabc6ce1727c178c59ff358db85a05f4fb1bd5e7e7f669143faad61ccddee65"} Oct 30 22:33:32.020618 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:32.020567 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0a2577-7f97948b44-d669v" event={"ID":"7365fe23-d73c-467d-bd8b-d0177976452f","Type":"ContainerDied","Data":"c4048ca82a867cfb8685bad7dacb13d9116a1f0ed6c96402c5ee3ff0d366c99b"} Oct 30 22:33:32.020618 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:32.020595 6720 scope.go:122] "RemoveContainer" containerID="6aabc6ce1727c178c59ff358db85a05f4fb1bd5e7e7f669143faad61ccddee65" Oct 30 22:33:32.032517 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:32.032255 6720 scope.go:122] "RemoveContainer" containerID="6aabc6ce1727c178c59ff358db85a05f4fb1bd5e7e7f669143faad61ccddee65" Oct 30 22:33:32.033689 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:33:32.032711 6720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6aabc6ce1727c178c59ff358db85a05f4fb1bd5e7e7f669143faad61ccddee65\": container with ID starting with 6aabc6ce1727c178c59ff358db85a05f4fb1bd5e7e7f669143faad61ccddee65 not found: ID does not exist" containerID="6aabc6ce1727c178c59ff358db85a05f4fb1bd5e7e7f669143faad61ccddee65" Oct 30 22:33:32.033689 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:32.032754 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6aabc6ce1727c178c59ff358db85a05f4fb1bd5e7e7f669143faad61ccddee65"} err="failed to get container status \"6aabc6ce1727c178c59ff358db85a05f4fb1bd5e7e7f669143faad61ccddee65\": rpc error: code = NotFound desc = could not find container \"6aabc6ce1727c178c59ff358db85a05f4fb1bd5e7e7f669143faad61ccddee65\": container with ID starting with 6aabc6ce1727c178c59ff358db85a05f4fb1bd5e7e7f669143faad61ccddee65 not found: ID does not exist" Oct 30 22:33:32.040414 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:32.040371 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0a2577-7f97948b44-d669v"] Oct 30 22:33:32.047079 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:32.047055 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-0a2577-7f97948b44-d669v"] Oct 30 22:33:32.445576 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:32.445527 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:33.117953 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:33.117372 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:33.321198 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:33.321170 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:33.482464 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:33.472536 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:33.888925 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:33.888894 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:33.958056 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:33.958023 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7365fe23-d73c-467d-bd8b-d0177976452f" path="/var/lib/kubelet/pods/7365fe23-d73c-467d-bd8b-d0177976452f/volumes" Oct 30 22:33:34.032479 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:34.032446 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:34.585883 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:34.585843 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:35.202901 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:35.202866 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:35.833692 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:35.833657 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:36.556326 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:36.556264 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:37.064999 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:37.064966 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f60026-564db895d5-nljrz"] Oct 30 22:33:37.067114 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:37.067083 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f60026-564db895d5-nljrz" podUID="0e1f3f99-8520-4097-b948-3ca747b75ec9" containerName="maas-api" containerID="cri-o://0d0c7f42eb53f8f3598c1ddb8216787a9dabcf85abd0dd340b02d9aedad9c7ef" gracePeriod=30 Oct 30 22:33:37.202551 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:37.202512 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:37.475682 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:37.475653 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:37.575323 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:37.575290 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f60026-564db895d5-nljrz" Oct 30 22:33:37.619951 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:37.619843 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:37.720676 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:37.719403 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e1f3f99-8520-4097-b948-3ca747b75ec9-kube-api-access-x9twd\" (UniqueName: \"kubernetes.io/projected/0e1f3f99-8520-4097-b948-3ca747b75ec9-kube-api-access-x9twd\") pod \"0e1f3f99-8520-4097-b948-3ca747b75ec9\" (UID: \"0e1f3f99-8520-4097-b948-3ca747b75ec9\") " Oct 30 22:33:37.720676 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:37.719589 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e1f3f99-8520-4097-b948-3ca747b75ec9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e1f3f99-8520-4097-b948-3ca747b75ec9-metrics-certs\") pod \"0e1f3f99-8520-4097-b948-3ca747b75ec9\" (UID: \"0e1f3f99-8520-4097-b948-3ca747b75ec9\") " Oct 30 22:33:37.720676 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:37.719648 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e1f3f99-8520-4097-b948-3ca747b75ec9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e1f3f99-8520-4097-b948-3ca747b75ec9-openshift-service-ca\") pod \"0e1f3f99-8520-4097-b948-3ca747b75ec9\" (UID: \"0e1f3f99-8520-4097-b948-3ca747b75ec9\") " Oct 30 22:33:37.720676 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:37.719682 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e1f3f99-8520-4097-b948-3ca747b75ec9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e1f3f99-8520-4097-b948-3ca747b75ec9-maas-api-tls\") pod \"0e1f3f99-8520-4097-b948-3ca747b75ec9\" (UID: \"0e1f3f99-8520-4097-b948-3ca747b75ec9\") " Oct 30 22:33:37.720676 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:37.720439 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e1f3f99-8520-4097-b948-3ca747b75ec9-openshift-service-ca" pod "0e1f3f99-8520-4097-b948-3ca747b75ec9" (UID: "0e1f3f99-8520-4097-b948-3ca747b75ec9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:33:37.723893 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:37.723869 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e1f3f99-8520-4097-b948-3ca747b75ec9-maas-api-tls" pod "0e1f3f99-8520-4097-b948-3ca747b75ec9" (UID: "0e1f3f99-8520-4097-b948-3ca747b75ec9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:33:37.727601 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:37.727566 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e1f3f99-8520-4097-b948-3ca747b75ec9-metrics-certs" pod "0e1f3f99-8520-4097-b948-3ca747b75ec9" (UID: "0e1f3f99-8520-4097-b948-3ca747b75ec9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:33:37.733465 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:37.733433 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e1f3f99-8520-4097-b948-3ca747b75ec9-kube-api-access-x9twd" pod "0e1f3f99-8520-4097-b948-3ca747b75ec9" (UID: "0e1f3f99-8520-4097-b948-3ca747b75ec9"). InnerVolumeSpecName "kube-api-access-x9twd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:33:37.752274 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:37.752226 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-9j9cx"] Oct 30 22:33:37.752560 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:37.752534 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-9j9cx" podUID="61ff25c3-567e-4fef-ad01-057108004032" containerName="istio-proxy" containerID="cri-o://e71465d6b128647e6f6e729a3c7ecdac7f28111d79208f8935a7a64832b5fa0d" gracePeriod=30 Oct 30 22:33:37.821615 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:37.821580 6720 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e1f3f99-8520-4097-b948-3ca747b75ec9-metrics-certs\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:37.821615 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:37.821610 6720 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e1f3f99-8520-4097-b948-3ca747b75ec9-openshift-service-ca\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:37.821615 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:37.821622 6720 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e1f3f99-8520-4097-b948-3ca747b75ec9-maas-api-tls\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:37.821925 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:37.821631 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9twd\" (UniqueName: \"kubernetes.io/projected/0e1f3f99-8520-4097-b948-3ca747b75ec9-kube-api-access-x9twd\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:37.860838 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:37.860803 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:38.069350 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:38.069319 6720 generic.go:356] "Generic (PLEG): container finished" podID="0e1f3f99-8520-4097-b948-3ca747b75ec9" containerID="0d0c7f42eb53f8f3598c1ddb8216787a9dabcf85abd0dd340b02d9aedad9c7ef" exitCode=0 Oct 30 22:33:38.069816 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:38.069402 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f60026-564db895d5-nljrz" event={"ID":"0e1f3f99-8520-4097-b948-3ca747b75ec9","Type":"ContainerDied","Data":"0d0c7f42eb53f8f3598c1ddb8216787a9dabcf85abd0dd340b02d9aedad9c7ef"} Oct 30 22:33:38.069816 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:38.069372 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f60026-564db895d5-nljrz" Oct 30 22:33:38.069816 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:38.069447 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f60026-564db895d5-nljrz" event={"ID":"0e1f3f99-8520-4097-b948-3ca747b75ec9","Type":"ContainerDied","Data":"69f1d640e19b55defd01ae7e5abdeaf68a3d8f56d04eb4ccfa17fe985a771bd1"} Oct 30 22:33:38.069816 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:38.069469 6720 scope.go:122] "RemoveContainer" containerID="0d0c7f42eb53f8f3598c1ddb8216787a9dabcf85abd0dd340b02d9aedad9c7ef" Oct 30 22:33:38.080454 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:38.080433 6720 scope.go:122] "RemoveContainer" containerID="0d0c7f42eb53f8f3598c1ddb8216787a9dabcf85abd0dd340b02d9aedad9c7ef" Oct 30 22:33:38.080756 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:33:38.080737 6720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d0c7f42eb53f8f3598c1ddb8216787a9dabcf85abd0dd340b02d9aedad9c7ef\": container with ID starting with 0d0c7f42eb53f8f3598c1ddb8216787a9dabcf85abd0dd340b02d9aedad9c7ef not found: ID does not exist" containerID="0d0c7f42eb53f8f3598c1ddb8216787a9dabcf85abd0dd340b02d9aedad9c7ef" Oct 30 22:33:38.080800 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:38.080769 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d0c7f42eb53f8f3598c1ddb8216787a9dabcf85abd0dd340b02d9aedad9c7ef"} err="failed to get container status \"0d0c7f42eb53f8f3598c1ddb8216787a9dabcf85abd0dd340b02d9aedad9c7ef\": rpc error: code = NotFound desc = could not find container \"0d0c7f42eb53f8f3598c1ddb8216787a9dabcf85abd0dd340b02d9aedad9c7ef\": container with ID starting with 0d0c7f42eb53f8f3598c1ddb8216787a9dabcf85abd0dd340b02d9aedad9c7ef not found: ID does not exist" Oct 30 22:33:38.089460 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:38.089366 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f60026-564db895d5-nljrz"] Oct 30 22:33:38.091455 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:38.091425 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f60026-564db895d5-nljrz"] Oct 30 22:33:38.097821 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:38.097794 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:38.633506 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:38.633462 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:39.193209 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:39.189599 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:39.347577 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:39.347539 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:39.505488 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:39.505328 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:39.649988 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:39.649926 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:39.956961 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:39.956923 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e1f3f99-8520-4097-b948-3ca747b75ec9" path="/var/lib/kubelet/pods/0e1f3f99-8520-4097-b948-3ca747b75ec9/volumes" Oct 30 22:33:40.016141 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:40.016109 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:40.173461 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:40.173412 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:40.406936 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:33:40.406897 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 22:33:41.211395 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:41.211325 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:41.930742 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:41.930699 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:42.701991 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:42.701933 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:43.057470 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:43.057446 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-9j9cx" Oct 30 22:33:43.113695 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:43.113658 6720 generic.go:356] "Generic (PLEG): container finished" podID="61ff25c3-567e-4fef-ad01-057108004032" containerID="e71465d6b128647e6f6e729a3c7ecdac7f28111d79208f8935a7a64832b5fa0d" exitCode=0 Oct 30 22:33:43.113887 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:43.113786 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-9j9cx" event={"ID":"61ff25c3-567e-4fef-ad01-057108004032","Type":"ContainerDied","Data":"e71465d6b128647e6f6e729a3c7ecdac7f28111d79208f8935a7a64832b5fa0d"} Oct 30 22:33:43.113887 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:43.113847 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-9j9cx" event={"ID":"61ff25c3-567e-4fef-ad01-057108004032","Type":"ContainerDied","Data":"edca9c7e640e7c9f6863b6f00156bb16de01cccefe922f00708d2aac61172234"} Oct 30 22:33:43.113887 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:43.113878 6720 scope.go:122] "RemoveContainer" containerID="e71465d6b128647e6f6e729a3c7ecdac7f28111d79208f8935a7a64832b5fa0d" Oct 30 22:33:43.114070 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:43.114019 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-9j9cx" Oct 30 22:33:43.124510 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:43.124483 6720 scope.go:122] "RemoveContainer" containerID="e71465d6b128647e6f6e729a3c7ecdac7f28111d79208f8935a7a64832b5fa0d" Oct 30 22:33:43.124865 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:33:43.124844 6720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e71465d6b128647e6f6e729a3c7ecdac7f28111d79208f8935a7a64832b5fa0d\": container with ID starting with e71465d6b128647e6f6e729a3c7ecdac7f28111d79208f8935a7a64832b5fa0d not found: ID does not exist" containerID="e71465d6b128647e6f6e729a3c7ecdac7f28111d79208f8935a7a64832b5fa0d" Oct 30 22:33:43.124974 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:43.124875 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e71465d6b128647e6f6e729a3c7ecdac7f28111d79208f8935a7a64832b5fa0d"} err="failed to get container status \"e71465d6b128647e6f6e729a3c7ecdac7f28111d79208f8935a7a64832b5fa0d\": rpc error: code = NotFound desc = could not find container \"e71465d6b128647e6f6e729a3c7ecdac7f28111d79208f8935a7a64832b5fa0d\": container with ID starting with e71465d6b128647e6f6e729a3c7ecdac7f28111d79208f8935a7a64832b5fa0d not found: ID does not exist" Oct 30 22:33:43.186667 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:43.186616 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/61ff25c3-567e-4fef-ad01-057108004032-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61ff25c3-567e-4fef-ad01-057108004032-istio-podinfo\") pod \"61ff25c3-567e-4fef-ad01-057108004032\" (UID: \"61ff25c3-567e-4fef-ad01-057108004032\") " Oct 30 22:33:43.186832 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:43.186716 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61ff25c3-567e-4fef-ad01-057108004032-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61ff25c3-567e-4fef-ad01-057108004032-workload-socket\") pod \"61ff25c3-567e-4fef-ad01-057108004032\" (UID: \"61ff25c3-567e-4fef-ad01-057108004032\") " Oct 30 22:33:43.186832 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:43.186766 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61ff25c3-567e-4fef-ad01-057108004032-kube-api-access-5g5jm\" (UniqueName: \"kubernetes.io/projected/61ff25c3-567e-4fef-ad01-057108004032-kube-api-access-5g5jm\") pod \"61ff25c3-567e-4fef-ad01-057108004032\" (UID: \"61ff25c3-567e-4fef-ad01-057108004032\") " Oct 30 22:33:43.186832 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:43.186783 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61ff25c3-567e-4fef-ad01-057108004032-istio-token\" (UniqueName: \"kubernetes.io/projected/61ff25c3-567e-4fef-ad01-057108004032-istio-token\") pod \"61ff25c3-567e-4fef-ad01-057108004032\" (UID: \"61ff25c3-567e-4fef-ad01-057108004032\") " Oct 30 22:33:43.186832 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:43.186822 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61ff25c3-567e-4fef-ad01-057108004032-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61ff25c3-567e-4fef-ad01-057108004032-workload-certs\") pod \"61ff25c3-567e-4fef-ad01-057108004032\" (UID: \"61ff25c3-567e-4fef-ad01-057108004032\") " Oct 30 22:33:43.187058 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:43.186871 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61ff25c3-567e-4fef-ad01-057108004032-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61ff25c3-567e-4fef-ad01-057108004032-istiod-ca-cert\") pod \"61ff25c3-567e-4fef-ad01-057108004032\" (UID: \"61ff25c3-567e-4fef-ad01-057108004032\") " Oct 30 22:33:43.187058 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:43.186969 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61ff25c3-567e-4fef-ad01-057108004032-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61ff25c3-567e-4fef-ad01-057108004032-istio-envoy\") pod \"61ff25c3-567e-4fef-ad01-057108004032\" (UID: \"61ff25c3-567e-4fef-ad01-057108004032\") " Oct 30 22:33:43.187058 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:43.186999 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61ff25c3-567e-4fef-ad01-057108004032-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61ff25c3-567e-4fef-ad01-057108004032-credential-socket\") pod \"61ff25c3-567e-4fef-ad01-057108004032\" (UID: \"61ff25c3-567e-4fef-ad01-057108004032\") " Oct 30 22:33:43.187058 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:43.187017 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61ff25c3-567e-4fef-ad01-057108004032-workload-socket" pod "61ff25c3-567e-4fef-ad01-057108004032" (UID: "61ff25c3-567e-4fef-ad01-057108004032"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:33:43.187058 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:43.187027 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61ff25c3-567e-4fef-ad01-057108004032-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61ff25c3-567e-4fef-ad01-057108004032-istio-data\") pod \"61ff25c3-567e-4fef-ad01-057108004032\" (UID: \"61ff25c3-567e-4fef-ad01-057108004032\") " Oct 30 22:33:43.187332 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:43.187178 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61ff25c3-567e-4fef-ad01-057108004032-workload-certs" pod "61ff25c3-567e-4fef-ad01-057108004032" (UID: "61ff25c3-567e-4fef-ad01-057108004032"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:33:43.187332 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:43.187228 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61ff25c3-567e-4fef-ad01-057108004032-istio-data" pod "61ff25c3-567e-4fef-ad01-057108004032" (UID: "61ff25c3-567e-4fef-ad01-057108004032"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:33:43.187478 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:43.187427 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61ff25c3-567e-4fef-ad01-057108004032-credential-socket" pod "61ff25c3-567e-4fef-ad01-057108004032" (UID: "61ff25c3-567e-4fef-ad01-057108004032"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:33:43.187660 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:43.187638 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61ff25c3-567e-4fef-ad01-057108004032-istiod-ca-cert" pod "61ff25c3-567e-4fef-ad01-057108004032" (UID: "61ff25c3-567e-4fef-ad01-057108004032"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:33:43.187937 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:43.187836 6720 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61ff25c3-567e-4fef-ad01-057108004032-workload-socket\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:43.187937 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:43.187867 6720 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61ff25c3-567e-4fef-ad01-057108004032-workload-certs\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:43.187937 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:43.187884 6720 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61ff25c3-567e-4fef-ad01-057108004032-istiod-ca-cert\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:43.187937 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:43.187898 6720 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61ff25c3-567e-4fef-ad01-057108004032-credential-socket\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:43.187937 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:43.187912 6720 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61ff25c3-567e-4fef-ad01-057108004032-istio-data\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:43.189357 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:43.189329 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/61ff25c3-567e-4fef-ad01-057108004032-istio-podinfo" pod "61ff25c3-567e-4fef-ad01-057108004032" (UID: "61ff25c3-567e-4fef-ad01-057108004032"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:33:43.189357 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:43.189349 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61ff25c3-567e-4fef-ad01-057108004032-kube-api-access-5g5jm" pod "61ff25c3-567e-4fef-ad01-057108004032" (UID: "61ff25c3-567e-4fef-ad01-057108004032"). InnerVolumeSpecName "kube-api-access-5g5jm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:33:43.189565 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:43.189378 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61ff25c3-567e-4fef-ad01-057108004032-istio-token" pod "61ff25c3-567e-4fef-ad01-057108004032" (UID: "61ff25c3-567e-4fef-ad01-057108004032"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:33:43.189565 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:43.189485 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61ff25c3-567e-4fef-ad01-057108004032-istio-envoy" pod "61ff25c3-567e-4fef-ad01-057108004032" (UID: "61ff25c3-567e-4fef-ad01-057108004032"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:33:43.289570 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:43.289496 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5g5jm\" (UniqueName: \"kubernetes.io/projected/61ff25c3-567e-4fef-ad01-057108004032-kube-api-access-5g5jm\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:43.289570 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:43.289524 6720 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61ff25c3-567e-4fef-ad01-057108004032-istio-token\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:43.289570 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:43.289534 6720 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61ff25c3-567e-4fef-ad01-057108004032-istio-envoy\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:43.289570 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:43.289542 6720 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61ff25c3-567e-4fef-ad01-057108004032-istio-podinfo\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:43.329776 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:43.329605 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:43.439789 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:43.439756 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-9j9cx"] Oct 30 22:33:43.443934 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:43.443908 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cca1c7-openshift-default-56cc7b4b8f-9j9cx"] Oct 30 22:33:43.956746 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:43.956716 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61ff25c3-567e-4fef-ad01-057108004032" path="/var/lib/kubelet/pods/61ff25c3-567e-4fef-ad01-057108004032/volumes" Oct 30 22:33:44.007453 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:44.005727 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:44.670440 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:44.670377 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:44.828521 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:44.827966 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:44.970545 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:44.970458 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1529dc/distinct-2-w0-1529dc-kserve-7788894c6-7mbrr"] Oct 30 22:33:45.009093 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:45.009053 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1529dc/simulator-w0-1529dc-kserve-8466d8d645-l54cf"] Oct 30 22:33:45.063052 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:45.063013 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:45.256260 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:45.256169 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:45.785535 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:45.785498 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:46.701329 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:46.701259 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:47.410480 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:47.410351 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:48.027883 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:48.027846 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:48.569935 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:48.569906 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:48.712940 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:48.712879 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:48.861214 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:48.861173 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:49.414839 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:49.414797 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:50.029516 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:50.029272 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:50.445091 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:50.445051 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:50.587022 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:50.586983 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:50.762062 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:50.761967 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:51.207593 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:51.207556 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0f73c9-88bff4c9f-qd4gf"] Oct 30 22:33:51.209512 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:51.209471 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0f73c9-88bff4c9f-qd4gf" podUID="f51bd7b2-4d7e-42af-9dbc-4083db247bf4" containerName="maas-api" containerID="cri-o://476330ecbd910a3b75ce1fc6afde42532023b9a4051aaea76c6e4eb032b213e5" gracePeriod=30 Oct 30 22:33:51.293750 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:51.293720 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:51.473871 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:51.473739 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:51.627153 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:51.627122 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:51.642474 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:51.642450 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0f73c9-88bff4c9f-qd4gf" Oct 30 22:33:51.701704 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:51.701668 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f51bd7b2-4d7e-42af-9dbc-4083db247bf4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f51bd7b2-4d7e-42af-9dbc-4083db247bf4-metrics-certs\") pod \"f51bd7b2-4d7e-42af-9dbc-4083db247bf4\" (UID: \"f51bd7b2-4d7e-42af-9dbc-4083db247bf4\") " Oct 30 22:33:51.701910 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:51.701788 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f51bd7b2-4d7e-42af-9dbc-4083db247bf4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f51bd7b2-4d7e-42af-9dbc-4083db247bf4-maas-api-tls\") pod \"f51bd7b2-4d7e-42af-9dbc-4083db247bf4\" (UID: \"f51bd7b2-4d7e-42af-9dbc-4083db247bf4\") " Oct 30 22:33:51.701910 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:51.701857 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f51bd7b2-4d7e-42af-9dbc-4083db247bf4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f51bd7b2-4d7e-42af-9dbc-4083db247bf4-openshift-service-ca\") pod \"f51bd7b2-4d7e-42af-9dbc-4083db247bf4\" (UID: \"f51bd7b2-4d7e-42af-9dbc-4083db247bf4\") " Oct 30 22:33:51.702055 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:51.701984 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f51bd7b2-4d7e-42af-9dbc-4083db247bf4-kube-api-access-gpwkn\" (UniqueName: \"kubernetes.io/projected/f51bd7b2-4d7e-42af-9dbc-4083db247bf4-kube-api-access-gpwkn\") pod \"f51bd7b2-4d7e-42af-9dbc-4083db247bf4\" (UID: \"f51bd7b2-4d7e-42af-9dbc-4083db247bf4\") " Oct 30 22:33:51.702277 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:51.702244 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f51bd7b2-4d7e-42af-9dbc-4083db247bf4-openshift-service-ca" pod "f51bd7b2-4d7e-42af-9dbc-4083db247bf4" (UID: "f51bd7b2-4d7e-42af-9dbc-4083db247bf4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:33:51.702504 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:51.702488 6720 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f51bd7b2-4d7e-42af-9dbc-4083db247bf4-openshift-service-ca\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:51.704333 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:51.704305 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f51bd7b2-4d7e-42af-9dbc-4083db247bf4-kube-api-access-gpwkn" pod "f51bd7b2-4d7e-42af-9dbc-4083db247bf4" (UID: "f51bd7b2-4d7e-42af-9dbc-4083db247bf4"). InnerVolumeSpecName "kube-api-access-gpwkn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:33:51.704617 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:51.704597 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f51bd7b2-4d7e-42af-9dbc-4083db247bf4-metrics-certs" pod "f51bd7b2-4d7e-42af-9dbc-4083db247bf4" (UID: "f51bd7b2-4d7e-42af-9dbc-4083db247bf4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:33:51.704791 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:51.704764 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f51bd7b2-4d7e-42af-9dbc-4083db247bf4-maas-api-tls" pod "f51bd7b2-4d7e-42af-9dbc-4083db247bf4" (UID: "f51bd7b2-4d7e-42af-9dbc-4083db247bf4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:33:51.803467 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:51.803363 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpwkn\" (UniqueName: \"kubernetes.io/projected/f51bd7b2-4d7e-42af-9dbc-4083db247bf4-kube-api-access-gpwkn\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:51.803467 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:51.803434 6720 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f51bd7b2-4d7e-42af-9dbc-4083db247bf4-metrics-certs\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:51.803467 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:51.803452 6720 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f51bd7b2-4d7e-42af-9dbc-4083db247bf4-maas-api-tls\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:33:51.864702 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:51.864667 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:52.126480 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:52.126436 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:52.217917 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:52.217876 6720 generic.go:356] "Generic (PLEG): container finished" podID="f51bd7b2-4d7e-42af-9dbc-4083db247bf4" containerID="476330ecbd910a3b75ce1fc6afde42532023b9a4051aaea76c6e4eb032b213e5" exitCode=0 Oct 30 22:33:52.218428 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:52.217932 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0f73c9-88bff4c9f-qd4gf" Oct 30 22:33:52.218428 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:52.217955 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0f73c9-88bff4c9f-qd4gf" event={"ID":"f51bd7b2-4d7e-42af-9dbc-4083db247bf4","Type":"ContainerDied","Data":"476330ecbd910a3b75ce1fc6afde42532023b9a4051aaea76c6e4eb032b213e5"} Oct 30 22:33:52.218428 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:52.218008 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0f73c9-88bff4c9f-qd4gf" event={"ID":"f51bd7b2-4d7e-42af-9dbc-4083db247bf4","Type":"ContainerDied","Data":"f7075771fc8b2ab6531751528821614944154144de9b42c4d7ec122327d97800"} Oct 30 22:33:52.218428 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:52.218033 6720 scope.go:122] "RemoveContainer" containerID="476330ecbd910a3b75ce1fc6afde42532023b9a4051aaea76c6e4eb032b213e5" Oct 30 22:33:52.227731 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:52.227710 6720 scope.go:122] "RemoveContainer" containerID="476330ecbd910a3b75ce1fc6afde42532023b9a4051aaea76c6e4eb032b213e5" Oct 30 22:33:52.228065 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:33:52.228043 6720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"476330ecbd910a3b75ce1fc6afde42532023b9a4051aaea76c6e4eb032b213e5\": container with ID starting with 476330ecbd910a3b75ce1fc6afde42532023b9a4051aaea76c6e4eb032b213e5 not found: ID does not exist" containerID="476330ecbd910a3b75ce1fc6afde42532023b9a4051aaea76c6e4eb032b213e5" Oct 30 22:33:52.228137 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:52.228076 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"476330ecbd910a3b75ce1fc6afde42532023b9a4051aaea76c6e4eb032b213e5"} err="failed to get container status \"476330ecbd910a3b75ce1fc6afde42532023b9a4051aaea76c6e4eb032b213e5\": rpc error: code = NotFound desc = could not find container \"476330ecbd910a3b75ce1fc6afde42532023b9a4051aaea76c6e4eb032b213e5\": container with ID starting with 476330ecbd910a3b75ce1fc6afde42532023b9a4051aaea76c6e4eb032b213e5 not found: ID does not exist" Oct 30 22:33:52.244175 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:52.244149 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0f73c9-88bff4c9f-qd4gf"] Oct 30 22:33:52.248636 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:52.248611 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-0f73c9-88bff4c9f-qd4gf"] Oct 30 22:33:52.648262 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:52.648219 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:53.257332 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:53.255518 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:53.931122 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:53.931084 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:53.958963 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:53.958928 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f51bd7b2-4d7e-42af-9dbc-4083db247bf4" path="/var/lib/kubelet/pods/f51bd7b2-4d7e-42af-9dbc-4083db247bf4/volumes" Oct 30 22:33:54.646097 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:54.646064 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:55.324540 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:55.324502 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:55.776375 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:55.776337 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:55.939929 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:55.939886 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:56.204764 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:56.204701 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:56.835073 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:56.835033 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:57.656664 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:57.656627 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:58.218512 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:58.218479 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:58.837928 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:58.837889 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:58.868174 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:58.868138 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grgv"] Oct 30 22:33:58.868440 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:58.868415 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grgv" podUID="9200859d-b207-4764-a75c-270ba29b4155" containerName="istio-proxy" containerID="cri-o://9a24b720f4b35ea65db5cf9b4d366e1758d35bb5ba5fb4bc716d0e760d1f9dde" gracePeriod=30 Oct 30 22:33:59.179558 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:59.179494 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:59.336833 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:59.336763 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:59.478110 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:59.478025 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:59.630431 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:59.627916 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:59.773896 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:59.773708 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:59.987862 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:59.987819 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:33:59.992499 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:33:59.992461 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-1529dc/distinct-2-w0-1529dc-kserve-7788894c6-7mbrr" podUID="09662f19-1ef9-406c-85ef-e402505875b4" containerName="main" containerID="cri-o://d3ec372bef2802692497034c99efdf7ae1b9b48ee8f973237376d07ff6f04247" gracePeriod=45 Oct 30 22:34:00.031423 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:00.031332 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-1529dc/simulator-w0-1529dc-kserve-8466d8d645-l54cf" podUID="257edb08-2e86-4f49-be57-848ace5958b1" containerName="main" containerID="cri-o://741f2b751312455dd9ef9940876e36b70c91bd5c78d59a00ce290bd0014f4a02" gracePeriod=45 Oct 30 22:34:00.135571 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:00.135532 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:00.285370 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:00.285163 6720 generic.go:356] "Generic (PLEG): container finished" podID="257edb08-2e86-4f49-be57-848ace5958b1" containerID="741f2b751312455dd9ef9940876e36b70c91bd5c78d59a00ce290bd0014f4a02" exitCode=0 Oct 30 22:34:00.285560 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:00.285248 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1529dc/simulator-w0-1529dc-kserve-8466d8d645-l54cf" event={"ID":"257edb08-2e86-4f49-be57-848ace5958b1","Type":"ContainerDied","Data":"741f2b751312455dd9ef9940876e36b70c91bd5c78d59a00ce290bd0014f4a02"} Oct 30 22:34:00.287116 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:00.287092 6720 generic.go:356] "Generic (PLEG): container finished" podID="09662f19-1ef9-406c-85ef-e402505875b4" containerID="d3ec372bef2802692497034c99efdf7ae1b9b48ee8f973237376d07ff6f04247" exitCode=0 Oct 30 22:34:00.287246 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:00.287172 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1529dc/distinct-2-w0-1529dc-kserve-7788894c6-7mbrr" event={"ID":"09662f19-1ef9-406c-85ef-e402505875b4","Type":"ContainerDied","Data":"d3ec372bef2802692497034c99efdf7ae1b9b48ee8f973237376d07ff6f04247"} Oct 30 22:34:00.344765 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:00.342959 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1529dc/distinct-2-w0-1529dc-kserve-7788894c6-7mbrr" Oct 30 22:34:00.356952 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:00.356924 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1529dc/simulator-w0-1529dc-kserve-8466d8d645-l54cf" Oct 30 22:34:00.517254 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:00.517217 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/257edb08-2e86-4f49-be57-848ace5958b1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/257edb08-2e86-4f49-be57-848ace5958b1-dshm\") pod \"257edb08-2e86-4f49-be57-848ace5958b1\" (UID: \"257edb08-2e86-4f49-be57-848ace5958b1\") " Oct 30 22:34:00.517452 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:00.517348 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09662f19-1ef9-406c-85ef-e402505875b4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/09662f19-1ef9-406c-85ef-e402505875b4-dshm\") pod \"09662f19-1ef9-406c-85ef-e402505875b4\" (UID: \"09662f19-1ef9-406c-85ef-e402505875b4\") " Oct 30 22:34:00.517452 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:00.517412 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/257edb08-2e86-4f49-be57-848ace5958b1-kube-api-access-wpglj\" (UniqueName: \"kubernetes.io/projected/257edb08-2e86-4f49-be57-848ace5958b1-kube-api-access-wpglj\") pod \"257edb08-2e86-4f49-be57-848ace5958b1\" (UID: \"257edb08-2e86-4f49-be57-848ace5958b1\") " Oct 30 22:34:00.517566 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:00.517463 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/257edb08-2e86-4f49-be57-848ace5958b1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/257edb08-2e86-4f49-be57-848ace5958b1-tmp-dir\") pod \"257edb08-2e86-4f49-be57-848ace5958b1\" (UID: \"257edb08-2e86-4f49-be57-848ace5958b1\") " Oct 30 22:34:00.517566 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:00.517502 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09662f19-1ef9-406c-85ef-e402505875b4-tls-certs\" (UniqueName: \"kubernetes.io/secret/09662f19-1ef9-406c-85ef-e402505875b4-tls-certs\") pod \"09662f19-1ef9-406c-85ef-e402505875b4\" (UID: \"09662f19-1ef9-406c-85ef-e402505875b4\") " Oct 30 22:34:00.517566 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:00.517546 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09662f19-1ef9-406c-85ef-e402505875b4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09662f19-1ef9-406c-85ef-e402505875b4-tmp-dir\") pod \"09662f19-1ef9-406c-85ef-e402505875b4\" (UID: \"09662f19-1ef9-406c-85ef-e402505875b4\") " Oct 30 22:34:00.517728 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:00.517635 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/257edb08-2e86-4f49-be57-848ace5958b1-home\" (UniqueName: \"kubernetes.io/empty-dir/257edb08-2e86-4f49-be57-848ace5958b1-home\") pod \"257edb08-2e86-4f49-be57-848ace5958b1\" (UID: \"257edb08-2e86-4f49-be57-848ace5958b1\") " Oct 30 22:34:00.517728 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:00.517675 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09662f19-1ef9-406c-85ef-e402505875b4-kube-api-access-w4h65\" (UniqueName: \"kubernetes.io/projected/09662f19-1ef9-406c-85ef-e402505875b4-kube-api-access-w4h65\") pod \"09662f19-1ef9-406c-85ef-e402505875b4\" (UID: \"09662f19-1ef9-406c-85ef-e402505875b4\") " Oct 30 22:34:00.517728 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:00.517718 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09662f19-1ef9-406c-85ef-e402505875b4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09662f19-1ef9-406c-85ef-e402505875b4-model-cache\") pod \"09662f19-1ef9-406c-85ef-e402505875b4\" (UID: \"09662f19-1ef9-406c-85ef-e402505875b4\") " Oct 30 22:34:00.518668 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:00.517745 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/257edb08-2e86-4f49-be57-848ace5958b1-tls-certs\" (UniqueName: \"kubernetes.io/secret/257edb08-2e86-4f49-be57-848ace5958b1-tls-certs\") pod \"257edb08-2e86-4f49-be57-848ace5958b1\" (UID: \"257edb08-2e86-4f49-be57-848ace5958b1\") " Oct 30 22:34:00.518668 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:00.517770 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09662f19-1ef9-406c-85ef-e402505875b4-home\" (UniqueName: \"kubernetes.io/empty-dir/09662f19-1ef9-406c-85ef-e402505875b4-home\") pod \"09662f19-1ef9-406c-85ef-e402505875b4\" (UID: \"09662f19-1ef9-406c-85ef-e402505875b4\") " Oct 30 22:34:00.518668 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:00.517841 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/257edb08-2e86-4f49-be57-848ace5958b1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/257edb08-2e86-4f49-be57-848ace5958b1-model-cache\") pod \"257edb08-2e86-4f49-be57-848ace5958b1\" (UID: \"257edb08-2e86-4f49-be57-848ace5958b1\") " Oct 30 22:34:00.518668 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:00.517916 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09662f19-1ef9-406c-85ef-e402505875b4-tmp-dir" pod "09662f19-1ef9-406c-85ef-e402505875b4" (UID: "09662f19-1ef9-406c-85ef-e402505875b4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:00.518668 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:00.518037 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/257edb08-2e86-4f49-be57-848ace5958b1-tmp-dir" pod "257edb08-2e86-4f49-be57-848ace5958b1" (UID: "257edb08-2e86-4f49-be57-848ace5958b1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:00.518668 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:00.518355 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09662f19-1ef9-406c-85ef-e402505875b4-model-cache" pod "09662f19-1ef9-406c-85ef-e402505875b4" (UID: "09662f19-1ef9-406c-85ef-e402505875b4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:00.518668 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:00.518481 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/257edb08-2e86-4f49-be57-848ace5958b1-home" pod "257edb08-2e86-4f49-be57-848ace5958b1" (UID: "257edb08-2e86-4f49-be57-848ace5958b1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:00.518668 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:00.518558 6720 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09662f19-1ef9-406c-85ef-e402505875b4-model-cache\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:00.518668 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:00.518582 6720 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/257edb08-2e86-4f49-be57-848ace5958b1-tmp-dir\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:00.518668 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:00.518599 6720 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09662f19-1ef9-406c-85ef-e402505875b4-tmp-dir\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:00.519174 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:00.518717 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09662f19-1ef9-406c-85ef-e402505875b4-home" pod "09662f19-1ef9-406c-85ef-e402505875b4" (UID: "09662f19-1ef9-406c-85ef-e402505875b4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:00.519174 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:00.518830 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/257edb08-2e86-4f49-be57-848ace5958b1-model-cache" pod "257edb08-2e86-4f49-be57-848ace5958b1" (UID: "257edb08-2e86-4f49-be57-848ace5958b1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:00.520461 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:00.520431 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/257edb08-2e86-4f49-be57-848ace5958b1-dshm" pod "257edb08-2e86-4f49-be57-848ace5958b1" (UID: "257edb08-2e86-4f49-be57-848ace5958b1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:00.520614 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:00.520567 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/257edb08-2e86-4f49-be57-848ace5958b1-kube-api-access-wpglj" pod "257edb08-2e86-4f49-be57-848ace5958b1" (UID: "257edb08-2e86-4f49-be57-848ace5958b1"). InnerVolumeSpecName "kube-api-access-wpglj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:34:00.520690 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:00.520676 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09662f19-1ef9-406c-85ef-e402505875b4-dshm" pod "09662f19-1ef9-406c-85ef-e402505875b4" (UID: "09662f19-1ef9-406c-85ef-e402505875b4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:00.520916 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:00.520892 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09662f19-1ef9-406c-85ef-e402505875b4-kube-api-access-w4h65" pod "09662f19-1ef9-406c-85ef-e402505875b4" (UID: "09662f19-1ef9-406c-85ef-e402505875b4"). InnerVolumeSpecName "kube-api-access-w4h65". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:34:00.521426 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:00.521377 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09662f19-1ef9-406c-85ef-e402505875b4-tls-certs" pod "09662f19-1ef9-406c-85ef-e402505875b4" (UID: "09662f19-1ef9-406c-85ef-e402505875b4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:34:00.521776 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:00.521750 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/257edb08-2e86-4f49-be57-848ace5958b1-tls-certs" pod "257edb08-2e86-4f49-be57-848ace5958b1" (UID: "257edb08-2e86-4f49-be57-848ace5958b1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:34:00.619286 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:00.619251 6720 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09662f19-1ef9-406c-85ef-e402505875b4-dshm\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:00.619286 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:00.619283 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpglj\" (UniqueName: \"kubernetes.io/projected/257edb08-2e86-4f49-be57-848ace5958b1-kube-api-access-wpglj\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:00.619538 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:00.619294 6720 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09662f19-1ef9-406c-85ef-e402505875b4-tls-certs\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:00.619538 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:00.619304 6720 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/257edb08-2e86-4f49-be57-848ace5958b1-home\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:00.619538 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:00.619315 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4h65\" (UniqueName: \"kubernetes.io/projected/09662f19-1ef9-406c-85ef-e402505875b4-kube-api-access-w4h65\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:00.619538 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:00.619324 6720 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/257edb08-2e86-4f49-be57-848ace5958b1-tls-certs\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:00.619538 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:00.619334 6720 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09662f19-1ef9-406c-85ef-e402505875b4-home\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:00.619538 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:00.619343 6720 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/257edb08-2e86-4f49-be57-848ace5958b1-model-cache\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:00.619538 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:00.619351 6720 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/257edb08-2e86-4f49-be57-848ace5958b1-dshm\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:00.796874 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:00.796837 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:01.294789 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:01.294697 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1529dc/simulator-w0-1529dc-kserve-8466d8d645-l54cf" Oct 30 22:34:01.294976 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:01.294697 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1529dc/simulator-w0-1529dc-kserve-8466d8d645-l54cf" event={"ID":"257edb08-2e86-4f49-be57-848ace5958b1","Type":"ContainerDied","Data":"516f8e92dc61ceaa3d1867e38b6ea4ae17b9029b85c0543294711ab6740b88a6"} Oct 30 22:34:01.294976 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:01.294828 6720 scope.go:122] "RemoveContainer" containerID="741f2b751312455dd9ef9940876e36b70c91bd5c78d59a00ce290bd0014f4a02" Oct 30 22:34:01.296217 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:01.296202 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-1529dc/distinct-2-w0-1529dc-kserve-7788894c6-7mbrr" Oct 30 22:34:01.296217 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:01.296207 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-1529dc/distinct-2-w0-1529dc-kserve-7788894c6-7mbrr" event={"ID":"09662f19-1ef9-406c-85ef-e402505875b4","Type":"ContainerDied","Data":"0bae8776c1b7ba368eb85dc7af161beeaf62c542a322e7d3fc31cf954db9bd51"} Oct 30 22:34:01.305725 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:01.305701 6720 scope.go:122] "RemoveContainer" containerID="d3ec372bef2802692497034c99efdf7ae1b9b48ee8f973237376d07ff6f04247" Oct 30 22:34:01.306666 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:01.306632 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/distinct-2-w0-1529dc-kserve-7788894c6-7mbrr" podUID="09662f19-1ef9-406c-85ef-e402505875b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:34:01.314617 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:01.314584 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-1529dc/simulator-w0-1529dc-kserve-8466d8d645-l54cf" podUID="257edb08-2e86-4f49-be57-848ace5958b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:34:01.326756 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:01.326721 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1529dc/simulator-w0-1529dc-kserve-8466d8d645-l54cf"] Oct 30 22:34:01.333662 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:01.333639 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-1529dc/simulator-w0-1529dc-kserve-8466d8d645-l54cf"] Oct 30 22:34:01.347003 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:01.346968 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-1529dc/distinct-2-w0-1529dc-kserve-7788894c6-7mbrr"] Oct 30 22:34:01.351615 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:01.351592 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-1529dc/distinct-2-w0-1529dc-kserve-7788894c6-7mbrr"] Oct 30 22:34:01.512568 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:01.512507 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:01.956322 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:01.956240 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09662f19-1ef9-406c-85ef-e402505875b4" path="/var/lib/kubelet/pods/09662f19-1ef9-406c-85ef-e402505875b4/volumes" Oct 30 22:34:01.956722 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:01.956708 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="257edb08-2e86-4f49-be57-848ace5958b1" path="/var/lib/kubelet/pods/257edb08-2e86-4f49-be57-848ace5958b1/volumes" Oct 30 22:34:02.243414 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:02.243291 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:02.667699 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:02.667660 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-1529dc-7d95c6c9d-9t9mq"] Oct 30 22:34:02.668219 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:02.667909 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1529dc-7d95c6c9d-9t9mq" podUID="c9cfa5a4-4d17-41f4-985c-c119257e3720" containerName="maas-api" containerID="cri-o://d5afdc8d09368bf32f15a3c808da4a6f19bd77139c8c933648ec5d64eb8c84db" gracePeriod=30 Oct 30 22:34:02.761089 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:02.761049 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:03.020507 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:02.994428 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:03.152972 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:03.152944 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1529dc-7d95c6c9d-9t9mq" Oct 30 22:34:03.162661 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:03.162630 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:03.230832 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:03.230795 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9cfa5a4-4d17-41f4-985c-c119257e3720-kube-api-access-vfpf4\" (UniqueName: \"kubernetes.io/projected/c9cfa5a4-4d17-41f4-985c-c119257e3720-kube-api-access-vfpf4\") pod \"c9cfa5a4-4d17-41f4-985c-c119257e3720\" (UID: \"c9cfa5a4-4d17-41f4-985c-c119257e3720\") " Oct 30 22:34:03.231013 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:03.230932 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9cfa5a4-4d17-41f4-985c-c119257e3720-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9cfa5a4-4d17-41f4-985c-c119257e3720-maas-api-tls\") pod \"c9cfa5a4-4d17-41f4-985c-c119257e3720\" (UID: \"c9cfa5a4-4d17-41f4-985c-c119257e3720\") " Oct 30 22:34:03.231074 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:03.231058 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9cfa5a4-4d17-41f4-985c-c119257e3720-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9cfa5a4-4d17-41f4-985c-c119257e3720-openshift-service-ca\") pod \"c9cfa5a4-4d17-41f4-985c-c119257e3720\" (UID: \"c9cfa5a4-4d17-41f4-985c-c119257e3720\") " Oct 30 22:34:03.231135 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:03.231120 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9cfa5a4-4d17-41f4-985c-c119257e3720-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9cfa5a4-4d17-41f4-985c-c119257e3720-metrics-certs\") pod \"c9cfa5a4-4d17-41f4-985c-c119257e3720\" (UID: \"c9cfa5a4-4d17-41f4-985c-c119257e3720\") " Oct 30 22:34:03.231459 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:03.231431 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9cfa5a4-4d17-41f4-985c-c119257e3720-openshift-service-ca" pod "c9cfa5a4-4d17-41f4-985c-c119257e3720" (UID: "c9cfa5a4-4d17-41f4-985c-c119257e3720"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:34:03.231649 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:03.231629 6720 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9cfa5a4-4d17-41f4-985c-c119257e3720-openshift-service-ca\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:03.233316 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:03.233288 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9cfa5a4-4d17-41f4-985c-c119257e3720-metrics-certs" pod "c9cfa5a4-4d17-41f4-985c-c119257e3720" (UID: "c9cfa5a4-4d17-41f4-985c-c119257e3720"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:34:03.233678 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:03.233658 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9cfa5a4-4d17-41f4-985c-c119257e3720-maas-api-tls" pod "c9cfa5a4-4d17-41f4-985c-c119257e3720" (UID: "c9cfa5a4-4d17-41f4-985c-c119257e3720"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:34:03.233748 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:03.233703 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9cfa5a4-4d17-41f4-985c-c119257e3720-kube-api-access-vfpf4" pod "c9cfa5a4-4d17-41f4-985c-c119257e3720" (UID: "c9cfa5a4-4d17-41f4-985c-c119257e3720"). InnerVolumeSpecName "kube-api-access-vfpf4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:34:03.314137 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:03.314020 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:03.320117 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:03.320060 6720 generic.go:356] "Generic (PLEG): container finished" podID="c9cfa5a4-4d17-41f4-985c-c119257e3720" containerID="d5afdc8d09368bf32f15a3c808da4a6f19bd77139c8c933648ec5d64eb8c84db" exitCode=0 Oct 30 22:34:03.320550 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:03.320521 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1529dc-7d95c6c9d-9t9mq" event={"ID":"c9cfa5a4-4d17-41f4-985c-c119257e3720","Type":"ContainerDied","Data":"d5afdc8d09368bf32f15a3c808da4a6f19bd77139c8c933648ec5d64eb8c84db"} Oct 30 22:34:03.320758 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:03.320738 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1529dc-7d95c6c9d-9t9mq" event={"ID":"c9cfa5a4-4d17-41f4-985c-c119257e3720","Type":"ContainerDied","Data":"e1d55f7857b0b02a074f5a20edfef3b36e0e918bf6de8f306b9749ffce8991eb"} Oct 30 22:34:03.323051 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:03.323021 6720 scope.go:122] "RemoveContainer" containerID="d5afdc8d09368bf32f15a3c808da4a6f19bd77139c8c933648ec5d64eb8c84db" Oct 30 22:34:03.323332 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:03.323313 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-1529dc-7d95c6c9d-9t9mq" Oct 30 22:34:03.332867 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:03.332829 6720 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9cfa5a4-4d17-41f4-985c-c119257e3720-metrics-certs\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:03.333063 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:03.333041 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfpf4\" (UniqueName: \"kubernetes.io/projected/c9cfa5a4-4d17-41f4-985c-c119257e3720-kube-api-access-vfpf4\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:03.333194 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:03.333176 6720 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9cfa5a4-4d17-41f4-985c-c119257e3720-maas-api-tls\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:03.358628 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:03.358296 6720 scope.go:122] "RemoveContainer" containerID="d5afdc8d09368bf32f15a3c808da4a6f19bd77139c8c933648ec5d64eb8c84db" Oct 30 22:34:03.359290 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:34:03.359183 6720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5afdc8d09368bf32f15a3c808da4a6f19bd77139c8c933648ec5d64eb8c84db\": container with ID starting with d5afdc8d09368bf32f15a3c808da4a6f19bd77139c8c933648ec5d64eb8c84db not found: ID does not exist" containerID="d5afdc8d09368bf32f15a3c808da4a6f19bd77139c8c933648ec5d64eb8c84db" Oct 30 22:34:03.359290 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:03.359225 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5afdc8d09368bf32f15a3c808da4a6f19bd77139c8c933648ec5d64eb8c84db"} err="failed to get container status \"d5afdc8d09368bf32f15a3c808da4a6f19bd77139c8c933648ec5d64eb8c84db\": rpc error: code = NotFound desc = could not find container \"d5afdc8d09368bf32f15a3c808da4a6f19bd77139c8c933648ec5d64eb8c84db\": container with ID starting with d5afdc8d09368bf32f15a3c808da4a6f19bd77139c8c933648ec5d64eb8c84db not found: ID does not exist" Oct 30 22:34:03.369034 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:03.368989 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-1529dc-7d95c6c9d-9t9mq"] Oct 30 22:34:03.373372 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:03.372458 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-1529dc-7d95c6c9d-9t9mq"] Oct 30 22:34:03.676856 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:03.676821 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:03.962913 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:03.962835 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9cfa5a4-4d17-41f4-985c-c119257e3720" path="/var/lib/kubelet/pods/c9cfa5a4-4d17-41f4-985c-c119257e3720/volumes" Oct 30 22:34:04.176813 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:04.176770 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grgv" Oct 30 22:34:04.243753 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:04.243661 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9200859d-b207-4764-a75c-270ba29b4155-istio-token\" (UniqueName: \"kubernetes.io/projected/9200859d-b207-4764-a75c-270ba29b4155-istio-token\") pod \"9200859d-b207-4764-a75c-270ba29b4155\" (UID: \"9200859d-b207-4764-a75c-270ba29b4155\") " Oct 30 22:34:04.243753 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:04.243708 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9200859d-b207-4764-a75c-270ba29b4155-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9200859d-b207-4764-a75c-270ba29b4155-istiod-ca-cert\") pod \"9200859d-b207-4764-a75c-270ba29b4155\" (UID: \"9200859d-b207-4764-a75c-270ba29b4155\") " Oct 30 22:34:04.243753 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:04.243745 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9200859d-b207-4764-a75c-270ba29b4155-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9200859d-b207-4764-a75c-270ba29b4155-workload-certs\") pod \"9200859d-b207-4764-a75c-270ba29b4155\" (UID: \"9200859d-b207-4764-a75c-270ba29b4155\") " Oct 30 22:34:04.244033 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:04.243764 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9200859d-b207-4764-a75c-270ba29b4155-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9200859d-b207-4764-a75c-270ba29b4155-workload-socket\") pod \"9200859d-b207-4764-a75c-270ba29b4155\" (UID: \"9200859d-b207-4764-a75c-270ba29b4155\") " Oct 30 22:34:04.244033 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:04.243798 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9200859d-b207-4764-a75c-270ba29b4155-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9200859d-b207-4764-a75c-270ba29b4155-istio-data\") pod \"9200859d-b207-4764-a75c-270ba29b4155\" (UID: \"9200859d-b207-4764-a75c-270ba29b4155\") " Oct 30 22:34:04.244033 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:04.243822 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9200859d-b207-4764-a75c-270ba29b4155-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9200859d-b207-4764-a75c-270ba29b4155-istio-envoy\") pod \"9200859d-b207-4764-a75c-270ba29b4155\" (UID: \"9200859d-b207-4764-a75c-270ba29b4155\") " Oct 30 22:34:04.244033 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:04.243879 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9200859d-b207-4764-a75c-270ba29b4155-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9200859d-b207-4764-a75c-270ba29b4155-istio-podinfo\") pod \"9200859d-b207-4764-a75c-270ba29b4155\" (UID: \"9200859d-b207-4764-a75c-270ba29b4155\") " Oct 30 22:34:04.244033 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:04.243929 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9200859d-b207-4764-a75c-270ba29b4155-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9200859d-b207-4764-a75c-270ba29b4155-credential-socket\") pod \"9200859d-b207-4764-a75c-270ba29b4155\" (UID: \"9200859d-b207-4764-a75c-270ba29b4155\") " Oct 30 22:34:04.244033 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:04.244022 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9200859d-b207-4764-a75c-270ba29b4155-kube-api-access-wl6zv\" (UniqueName: \"kubernetes.io/projected/9200859d-b207-4764-a75c-270ba29b4155-kube-api-access-wl6zv\") pod \"9200859d-b207-4764-a75c-270ba29b4155\" (UID: \"9200859d-b207-4764-a75c-270ba29b4155\") " Oct 30 22:34:04.244456 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:04.244121 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9200859d-b207-4764-a75c-270ba29b4155-workload-socket" pod "9200859d-b207-4764-a75c-270ba29b4155" (UID: "9200859d-b207-4764-a75c-270ba29b4155"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:04.244456 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:04.244224 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9200859d-b207-4764-a75c-270ba29b4155-istiod-ca-cert" pod "9200859d-b207-4764-a75c-270ba29b4155" (UID: "9200859d-b207-4764-a75c-270ba29b4155"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:34:04.244456 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:04.244308 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9200859d-b207-4764-a75c-270ba29b4155-istio-data" pod "9200859d-b207-4764-a75c-270ba29b4155" (UID: "9200859d-b207-4764-a75c-270ba29b4155"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:04.244658 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:04.244630 6720 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9200859d-b207-4764-a75c-270ba29b4155-istiod-ca-cert\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:04.244723 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:04.244663 6720 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9200859d-b207-4764-a75c-270ba29b4155-workload-socket\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:04.244723 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:04.244676 6720 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9200859d-b207-4764-a75c-270ba29b4155-istio-data\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:04.249616 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:04.249570 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9200859d-b207-4764-a75c-270ba29b4155-credential-socket" pod "9200859d-b207-4764-a75c-270ba29b4155" (UID: "9200859d-b207-4764-a75c-270ba29b4155"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:04.250989 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:04.250960 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9200859d-b207-4764-a75c-270ba29b4155-workload-certs" pod "9200859d-b207-4764-a75c-270ba29b4155" (UID: "9200859d-b207-4764-a75c-270ba29b4155"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:04.266806 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:04.266777 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9200859d-b207-4764-a75c-270ba29b4155-kube-api-access-wl6zv" pod "9200859d-b207-4764-a75c-270ba29b4155" (UID: "9200859d-b207-4764-a75c-270ba29b4155"). InnerVolumeSpecName "kube-api-access-wl6zv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:34:04.268365 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:04.268298 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9200859d-b207-4764-a75c-270ba29b4155-istio-envoy" pod "9200859d-b207-4764-a75c-270ba29b4155" (UID: "9200859d-b207-4764-a75c-270ba29b4155"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:04.268365 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:04.268306 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9200859d-b207-4764-a75c-270ba29b4155-istio-podinfo" pod "9200859d-b207-4764-a75c-270ba29b4155" (UID: "9200859d-b207-4764-a75c-270ba29b4155"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:34:04.268365 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:04.268330 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9200859d-b207-4764-a75c-270ba29b4155-istio-token" pod "9200859d-b207-4764-a75c-270ba29b4155" (UID: "9200859d-b207-4764-a75c-270ba29b4155"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:34:04.269718 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:04.269691 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:04.327891 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:04.327854 6720 generic.go:356] "Generic (PLEG): container finished" podID="9200859d-b207-4764-a75c-270ba29b4155" containerID="9a24b720f4b35ea65db5cf9b4d366e1758d35bb5ba5fb4bc716d0e760d1f9dde" exitCode=0 Oct 30 22:34:04.328076 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:04.327899 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grgv" Oct 30 22:34:04.328076 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:04.327941 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grgv" event={"ID":"9200859d-b207-4764-a75c-270ba29b4155","Type":"ContainerDied","Data":"9a24b720f4b35ea65db5cf9b4d366e1758d35bb5ba5fb4bc716d0e760d1f9dde"} Oct 30 22:34:04.328076 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:04.328002 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grgv" event={"ID":"9200859d-b207-4764-a75c-270ba29b4155","Type":"ContainerDied","Data":"9988e7971cd25c4c578cf2f72aa8fb4d74f90daa9bc013228150be0a65c99e11"} Oct 30 22:34:04.328076 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:04.328036 6720 scope.go:122] "RemoveContainer" containerID="9a24b720f4b35ea65db5cf9b4d366e1758d35bb5ba5fb4bc716d0e760d1f9dde" Oct 30 22:34:04.343806 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:04.343448 6720 scope.go:122] "RemoveContainer" containerID="9a24b720f4b35ea65db5cf9b4d366e1758d35bb5ba5fb4bc716d0e760d1f9dde" Oct 30 22:34:04.344454 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:34:04.344354 6720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a24b720f4b35ea65db5cf9b4d366e1758d35bb5ba5fb4bc716d0e760d1f9dde\": container with ID starting with 9a24b720f4b35ea65db5cf9b4d366e1758d35bb5ba5fb4bc716d0e760d1f9dde not found: ID does not exist" containerID="9a24b720f4b35ea65db5cf9b4d366e1758d35bb5ba5fb4bc716d0e760d1f9dde" Oct 30 22:34:04.344454 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:04.344412 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a24b720f4b35ea65db5cf9b4d366e1758d35bb5ba5fb4bc716d0e760d1f9dde"} err="failed to get container status \"9a24b720f4b35ea65db5cf9b4d366e1758d35bb5ba5fb4bc716d0e760d1f9dde\": rpc error: code = NotFound desc = could not find container \"9a24b720f4b35ea65db5cf9b4d366e1758d35bb5ba5fb4bc716d0e760d1f9dde\": container with ID starting with 9a24b720f4b35ea65db5cf9b4d366e1758d35bb5ba5fb4bc716d0e760d1f9dde not found: ID does not exist" Oct 30 22:34:04.346148 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:04.346120 6720 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9200859d-b207-4764-a75c-270ba29b4155-credential-socket\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:04.346283 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:04.346151 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wl6zv\" (UniqueName: \"kubernetes.io/projected/9200859d-b207-4764-a75c-270ba29b4155-kube-api-access-wl6zv\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:04.346283 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:04.346166 6720 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9200859d-b207-4764-a75c-270ba29b4155-istio-token\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:04.346283 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:04.346180 6720 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9200859d-b207-4764-a75c-270ba29b4155-workload-certs\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:04.346283 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:04.346193 6720 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9200859d-b207-4764-a75c-270ba29b4155-istio-envoy\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:04.346283 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:04.346205 6720 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9200859d-b207-4764-a75c-270ba29b4155-istio-podinfo\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:04.372090 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:04.372058 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grgv"] Oct 30 22:34:04.378454 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:04.378361 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-85544d4cd7-8grgv"] Oct 30 22:34:04.906935 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:04.906901 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-ceb36a/distinct-2-w3-ceb36a-kserve-7c59695467-46gdc"] Oct 30 22:34:04.924838 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:04.924808 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-ceb36a/premium-w3-ceb36a-kserve-6576bb476f-j5kdr"] Oct 30 22:34:05.037633 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:05.037595 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:05.252744 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:05.252663 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:05.937435 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:05.937402 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:05.957112 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:05.957080 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9200859d-b207-4764-a75c-270ba29b4155" path="/var/lib/kubelet/pods/9200859d-b207-4764-a75c-270ba29b4155/volumes" Oct 30 22:34:06.629978 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:06.629324 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:06.673473 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:06.673430 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-6brww"] Oct 30 22:34:06.673748 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:06.673723 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-6brww" podUID="8566ec43-9f32-4ad1-a46f-fc94ccf1b24d" containerName="istio-proxy" containerID="cri-o://c12034b9c83b2e5b8972284dde2537e89afc73effffc3b2df7b4672b876a74ad" gracePeriod=30 Oct 30 22:34:07.306943 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:07.306907 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:07.447956 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:07.447926 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:07.722414 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:07.716954 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:08.020399 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:08.020278 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:08.911410 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:08.911196 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:09.628942 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:09.628900 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:10.281911 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:10.281870 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:10.405466 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:34:10.405429 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 22:34:10.420623 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:10.420594 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:10.577630 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:10.577536 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:10.840467 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:10.837536 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-ljm7z"] Oct 30 22:34:10.840467 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:10.837791 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-ljm7z" podUID="919d285e-2e4f-41b9-95f8-87144df6a112" containerName="istio-proxy" containerID="cri-o://6b4e8d5d4fab470b21e36969ed58fe803ba5ccd055d4dd7ef8ceee78bdfe968c" gracePeriod=30 Oct 30 22:34:10.926873 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:10.926815 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:11.080243 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:11.080208 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:11.954020 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:11.953945 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-6brww" Oct 30 22:34:12.033980 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:12.033944 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:12.067067 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:12.067031 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8566ec43-9f32-4ad1-a46f-fc94ccf1b24d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8566ec43-9f32-4ad1-a46f-fc94ccf1b24d-credential-socket\") pod \"8566ec43-9f32-4ad1-a46f-fc94ccf1b24d\" (UID: \"8566ec43-9f32-4ad1-a46f-fc94ccf1b24d\") " Oct 30 22:34:12.067227 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:12.067089 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8566ec43-9f32-4ad1-a46f-fc94ccf1b24d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8566ec43-9f32-4ad1-a46f-fc94ccf1b24d-workload-certs\") pod \"8566ec43-9f32-4ad1-a46f-fc94ccf1b24d\" (UID: \"8566ec43-9f32-4ad1-a46f-fc94ccf1b24d\") " Oct 30 22:34:12.067227 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:12.067115 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8566ec43-9f32-4ad1-a46f-fc94ccf1b24d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8566ec43-9f32-4ad1-a46f-fc94ccf1b24d-workload-socket\") pod \"8566ec43-9f32-4ad1-a46f-fc94ccf1b24d\" (UID: \"8566ec43-9f32-4ad1-a46f-fc94ccf1b24d\") " Oct 30 22:34:12.067227 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:12.067138 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8566ec43-9f32-4ad1-a46f-fc94ccf1b24d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8566ec43-9f32-4ad1-a46f-fc94ccf1b24d-istio-podinfo\") pod \"8566ec43-9f32-4ad1-a46f-fc94ccf1b24d\" (UID: \"8566ec43-9f32-4ad1-a46f-fc94ccf1b24d\") " Oct 30 22:34:12.067227 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:12.067179 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8566ec43-9f32-4ad1-a46f-fc94ccf1b24d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8566ec43-9f32-4ad1-a46f-fc94ccf1b24d-istio-envoy\") pod \"8566ec43-9f32-4ad1-a46f-fc94ccf1b24d\" (UID: \"8566ec43-9f32-4ad1-a46f-fc94ccf1b24d\") " Oct 30 22:34:12.067227 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:12.067201 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8566ec43-9f32-4ad1-a46f-fc94ccf1b24d-istio-token\" (UniqueName: \"kubernetes.io/projected/8566ec43-9f32-4ad1-a46f-fc94ccf1b24d-istio-token\") pod \"8566ec43-9f32-4ad1-a46f-fc94ccf1b24d\" (UID: \"8566ec43-9f32-4ad1-a46f-fc94ccf1b24d\") " Oct 30 22:34:12.067564 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:12.067246 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8566ec43-9f32-4ad1-a46f-fc94ccf1b24d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8566ec43-9f32-4ad1-a46f-fc94ccf1b24d-istiod-ca-cert\") pod \"8566ec43-9f32-4ad1-a46f-fc94ccf1b24d\" (UID: \"8566ec43-9f32-4ad1-a46f-fc94ccf1b24d\") " Oct 30 22:34:12.067564 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:12.067283 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8566ec43-9f32-4ad1-a46f-fc94ccf1b24d-kube-api-access-w6czh\" (UniqueName: \"kubernetes.io/projected/8566ec43-9f32-4ad1-a46f-fc94ccf1b24d-kube-api-access-w6czh\") pod \"8566ec43-9f32-4ad1-a46f-fc94ccf1b24d\" (UID: \"8566ec43-9f32-4ad1-a46f-fc94ccf1b24d\") " Oct 30 22:34:12.067564 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:12.067366 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8566ec43-9f32-4ad1-a46f-fc94ccf1b24d-credential-socket" pod "8566ec43-9f32-4ad1-a46f-fc94ccf1b24d" (UID: "8566ec43-9f32-4ad1-a46f-fc94ccf1b24d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:12.067564 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:12.067378 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8566ec43-9f32-4ad1-a46f-fc94ccf1b24d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8566ec43-9f32-4ad1-a46f-fc94ccf1b24d-istio-data\") pod \"8566ec43-9f32-4ad1-a46f-fc94ccf1b24d\" (UID: \"8566ec43-9f32-4ad1-a46f-fc94ccf1b24d\") " Oct 30 22:34:12.067778 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:12.067427 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8566ec43-9f32-4ad1-a46f-fc94ccf1b24d-workload-certs" pod "8566ec43-9f32-4ad1-a46f-fc94ccf1b24d" (UID: "8566ec43-9f32-4ad1-a46f-fc94ccf1b24d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:12.067778 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:12.067632 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8566ec43-9f32-4ad1-a46f-fc94ccf1b24d-istio-data" pod "8566ec43-9f32-4ad1-a46f-fc94ccf1b24d" (UID: "8566ec43-9f32-4ad1-a46f-fc94ccf1b24d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:12.067931 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:12.067909 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8566ec43-9f32-4ad1-a46f-fc94ccf1b24d-workload-socket" pod "8566ec43-9f32-4ad1-a46f-fc94ccf1b24d" (UID: "8566ec43-9f32-4ad1-a46f-fc94ccf1b24d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:12.068263 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:12.067982 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8566ec43-9f32-4ad1-a46f-fc94ccf1b24d-istiod-ca-cert" pod "8566ec43-9f32-4ad1-a46f-fc94ccf1b24d" (UID: "8566ec43-9f32-4ad1-a46f-fc94ccf1b24d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:34:12.068263 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:12.068068 6720 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8566ec43-9f32-4ad1-a46f-fc94ccf1b24d-istio-data\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:12.068263 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:12.068089 6720 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8566ec43-9f32-4ad1-a46f-fc94ccf1b24d-credential-socket\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:12.068263 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:12.068103 6720 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8566ec43-9f32-4ad1-a46f-fc94ccf1b24d-workload-certs\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:12.068263 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:12.068117 6720 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8566ec43-9f32-4ad1-a46f-fc94ccf1b24d-workload-socket\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:12.070027 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:12.070000 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8566ec43-9f32-4ad1-a46f-fc94ccf1b24d-istio-envoy" pod "8566ec43-9f32-4ad1-a46f-fc94ccf1b24d" (UID: "8566ec43-9f32-4ad1-a46f-fc94ccf1b24d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:12.070129 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:12.070067 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8566ec43-9f32-4ad1-a46f-fc94ccf1b24d-istio-podinfo" pod "8566ec43-9f32-4ad1-a46f-fc94ccf1b24d" (UID: "8566ec43-9f32-4ad1-a46f-fc94ccf1b24d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:34:12.070225 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:12.070202 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8566ec43-9f32-4ad1-a46f-fc94ccf1b24d-istio-token" pod "8566ec43-9f32-4ad1-a46f-fc94ccf1b24d" (UID: "8566ec43-9f32-4ad1-a46f-fc94ccf1b24d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:34:12.070225 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:12.070203 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8566ec43-9f32-4ad1-a46f-fc94ccf1b24d-kube-api-access-w6czh" pod "8566ec43-9f32-4ad1-a46f-fc94ccf1b24d" (UID: "8566ec43-9f32-4ad1-a46f-fc94ccf1b24d"). InnerVolumeSpecName "kube-api-access-w6czh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:34:12.168763 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:12.168730 6720 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8566ec43-9f32-4ad1-a46f-fc94ccf1b24d-istio-podinfo\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:12.168763 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:12.168760 6720 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8566ec43-9f32-4ad1-a46f-fc94ccf1b24d-istio-envoy\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:12.168763 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:12.168769 6720 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8566ec43-9f32-4ad1-a46f-fc94ccf1b24d-istio-token\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:12.169018 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:12.168778 6720 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8566ec43-9f32-4ad1-a46f-fc94ccf1b24d-istiod-ca-cert\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:12.169018 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:12.168788 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6czh\" (UniqueName: \"kubernetes.io/projected/8566ec43-9f32-4ad1-a46f-fc94ccf1b24d-kube-api-access-w6czh\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:12.395424 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:12.395363 6720 generic.go:356] "Generic (PLEG): container finished" podID="8566ec43-9f32-4ad1-a46f-fc94ccf1b24d" containerID="c12034b9c83b2e5b8972284dde2537e89afc73effffc3b2df7b4672b876a74ad" exitCode=0 Oct 30 22:34:12.395590 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:12.395434 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-6brww" event={"ID":"8566ec43-9f32-4ad1-a46f-fc94ccf1b24d","Type":"ContainerDied","Data":"c12034b9c83b2e5b8972284dde2537e89afc73effffc3b2df7b4672b876a74ad"} Oct 30 22:34:12.395590 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:12.395445 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-6brww" Oct 30 22:34:12.395590 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:12.395481 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-6brww" event={"ID":"8566ec43-9f32-4ad1-a46f-fc94ccf1b24d","Type":"ContainerDied","Data":"55e43d638b333b18715136592b076adc9f8b6189672ae3c7d2a4ac859d5a5e59"} Oct 30 22:34:12.395590 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:12.395503 6720 scope.go:122] "RemoveContainer" containerID="c12034b9c83b2e5b8972284dde2537e89afc73effffc3b2df7b4672b876a74ad" Oct 30 22:34:12.406111 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:12.406090 6720 scope.go:122] "RemoveContainer" containerID="c12034b9c83b2e5b8972284dde2537e89afc73effffc3b2df7b4672b876a74ad" Oct 30 22:34:12.406415 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:34:12.406365 6720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c12034b9c83b2e5b8972284dde2537e89afc73effffc3b2df7b4672b876a74ad\": container with ID starting with c12034b9c83b2e5b8972284dde2537e89afc73effffc3b2df7b4672b876a74ad not found: ID does not exist" containerID="c12034b9c83b2e5b8972284dde2537e89afc73effffc3b2df7b4672b876a74ad" Oct 30 22:34:12.406485 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:12.406433 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c12034b9c83b2e5b8972284dde2537e89afc73effffc3b2df7b4672b876a74ad"} err="failed to get container status \"c12034b9c83b2e5b8972284dde2537e89afc73effffc3b2df7b4672b876a74ad\": rpc error: code = NotFound desc = could not find container \"c12034b9c83b2e5b8972284dde2537e89afc73effffc3b2df7b4672b876a74ad\": container with ID starting with c12034b9c83b2e5b8972284dde2537e89afc73effffc3b2df7b4672b876a74ad not found: ID does not exist" Oct 30 22:34:12.422605 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:12.422577 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-6brww"] Oct 30 22:34:12.427061 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:12.427036 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-a39158-openshift-default-575c648d9-6brww"] Oct 30 22:34:12.846640 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:12.846516 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:13.749568 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:13.749522 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:13.956644 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:13.956609 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8566ec43-9f32-4ad1-a46f-fc94ccf1b24d" path="/var/lib/kubelet/pods/8566ec43-9f32-4ad1-a46f-fc94ccf1b24d/volumes" Oct 30 22:34:14.542957 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:14.542924 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:15.432667 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:15.432630 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:16.168642 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:16.168552 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-ljm7z" Oct 30 22:34:16.275180 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:16.275144 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:16.313968 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:16.313934 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/919d285e-2e4f-41b9-95f8-87144df6a112-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/919d285e-2e4f-41b9-95f8-87144df6a112-istio-envoy\") pod \"919d285e-2e4f-41b9-95f8-87144df6a112\" (UID: \"919d285e-2e4f-41b9-95f8-87144df6a112\") " Oct 30 22:34:16.314133 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:16.313984 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/919d285e-2e4f-41b9-95f8-87144df6a112-kube-api-access-l4cpl\" (UniqueName: \"kubernetes.io/projected/919d285e-2e4f-41b9-95f8-87144df6a112-kube-api-access-l4cpl\") pod \"919d285e-2e4f-41b9-95f8-87144df6a112\" (UID: \"919d285e-2e4f-41b9-95f8-87144df6a112\") " Oct 30 22:34:16.314133 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:16.314013 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/919d285e-2e4f-41b9-95f8-87144df6a112-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/919d285e-2e4f-41b9-95f8-87144df6a112-istiod-ca-cert\") pod \"919d285e-2e4f-41b9-95f8-87144df6a112\" (UID: \"919d285e-2e4f-41b9-95f8-87144df6a112\") " Oct 30 22:34:16.314133 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:16.314050 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/919d285e-2e4f-41b9-95f8-87144df6a112-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/919d285e-2e4f-41b9-95f8-87144df6a112-istio-data\") pod \"919d285e-2e4f-41b9-95f8-87144df6a112\" (UID: \"919d285e-2e4f-41b9-95f8-87144df6a112\") " Oct 30 22:34:16.314133 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:16.314091 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/919d285e-2e4f-41b9-95f8-87144df6a112-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/919d285e-2e4f-41b9-95f8-87144df6a112-workload-certs\") pod \"919d285e-2e4f-41b9-95f8-87144df6a112\" (UID: \"919d285e-2e4f-41b9-95f8-87144df6a112\") " Oct 30 22:34:16.314348 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:16.314156 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/919d285e-2e4f-41b9-95f8-87144df6a112-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/919d285e-2e4f-41b9-95f8-87144df6a112-credential-socket\") pod \"919d285e-2e4f-41b9-95f8-87144df6a112\" (UID: \"919d285e-2e4f-41b9-95f8-87144df6a112\") " Oct 30 22:34:16.314348 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:16.314193 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/919d285e-2e4f-41b9-95f8-87144df6a112-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/919d285e-2e4f-41b9-95f8-87144df6a112-istio-podinfo\") pod \"919d285e-2e4f-41b9-95f8-87144df6a112\" (UID: \"919d285e-2e4f-41b9-95f8-87144df6a112\") " Oct 30 22:34:16.314348 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:16.314224 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/919d285e-2e4f-41b9-95f8-87144df6a112-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/919d285e-2e4f-41b9-95f8-87144df6a112-workload-socket\") pod \"919d285e-2e4f-41b9-95f8-87144df6a112\" (UID: \"919d285e-2e4f-41b9-95f8-87144df6a112\") " Oct 30 22:34:16.314348 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:16.314302 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/919d285e-2e4f-41b9-95f8-87144df6a112-istio-token\" (UniqueName: \"kubernetes.io/projected/919d285e-2e4f-41b9-95f8-87144df6a112-istio-token\") pod \"919d285e-2e4f-41b9-95f8-87144df6a112\" (UID: \"919d285e-2e4f-41b9-95f8-87144df6a112\") " Oct 30 22:34:16.314718 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:16.314457 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/919d285e-2e4f-41b9-95f8-87144df6a112-istio-data" pod "919d285e-2e4f-41b9-95f8-87144df6a112" (UID: "919d285e-2e4f-41b9-95f8-87144df6a112"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:16.314718 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:16.314479 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/919d285e-2e4f-41b9-95f8-87144df6a112-credential-socket" pod "919d285e-2e4f-41b9-95f8-87144df6a112" (UID: "919d285e-2e4f-41b9-95f8-87144df6a112"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:16.314718 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:16.314522 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/919d285e-2e4f-41b9-95f8-87144df6a112-workload-certs" pod "919d285e-2e4f-41b9-95f8-87144df6a112" (UID: "919d285e-2e4f-41b9-95f8-87144df6a112"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:16.314718 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:16.314540 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/919d285e-2e4f-41b9-95f8-87144df6a112-istiod-ca-cert" pod "919d285e-2e4f-41b9-95f8-87144df6a112" (UID: "919d285e-2e4f-41b9-95f8-87144df6a112"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:34:16.314718 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:16.314676 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/919d285e-2e4f-41b9-95f8-87144df6a112-workload-socket" pod "919d285e-2e4f-41b9-95f8-87144df6a112" (UID: "919d285e-2e4f-41b9-95f8-87144df6a112"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:16.315005 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:16.314863 6720 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/919d285e-2e4f-41b9-95f8-87144df6a112-istiod-ca-cert\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:16.315005 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:16.314876 6720 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/919d285e-2e4f-41b9-95f8-87144df6a112-istio-data\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:16.315005 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:16.314885 6720 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/919d285e-2e4f-41b9-95f8-87144df6a112-workload-certs\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:16.315005 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:16.314893 6720 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/919d285e-2e4f-41b9-95f8-87144df6a112-credential-socket\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:16.315005 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:16.314903 6720 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/919d285e-2e4f-41b9-95f8-87144df6a112-workload-socket\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:16.316547 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:16.316516 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/919d285e-2e4f-41b9-95f8-87144df6a112-kube-api-access-l4cpl" pod "919d285e-2e4f-41b9-95f8-87144df6a112" (UID: "919d285e-2e4f-41b9-95f8-87144df6a112"). InnerVolumeSpecName "kube-api-access-l4cpl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:34:16.316667 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:16.316559 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/919d285e-2e4f-41b9-95f8-87144df6a112-istio-envoy" pod "919d285e-2e4f-41b9-95f8-87144df6a112" (UID: "919d285e-2e4f-41b9-95f8-87144df6a112"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:16.316719 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:16.316703 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/919d285e-2e4f-41b9-95f8-87144df6a112-istio-token" pod "919d285e-2e4f-41b9-95f8-87144df6a112" (UID: "919d285e-2e4f-41b9-95f8-87144df6a112"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:34:16.316967 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:16.316950 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/919d285e-2e4f-41b9-95f8-87144df6a112-istio-podinfo" pod "919d285e-2e4f-41b9-95f8-87144df6a112" (UID: "919d285e-2e4f-41b9-95f8-87144df6a112"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:34:16.416123 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:16.416091 6720 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/919d285e-2e4f-41b9-95f8-87144df6a112-istio-envoy\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:16.416123 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:16.416119 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4cpl\" (UniqueName: \"kubernetes.io/projected/919d285e-2e4f-41b9-95f8-87144df6a112-kube-api-access-l4cpl\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:16.416123 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:16.416130 6720 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/919d285e-2e4f-41b9-95f8-87144df6a112-istio-podinfo\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:16.416370 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:16.416139 6720 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/919d285e-2e4f-41b9-95f8-87144df6a112-istio-token\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:16.418284 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:16.418252 6720 generic.go:356] "Generic (PLEG): container finished" podID="919d285e-2e4f-41b9-95f8-87144df6a112" containerID="6b4e8d5d4fab470b21e36969ed58fe803ba5ccd055d4dd7ef8ceee78bdfe968c" exitCode=0 Oct 30 22:34:16.418439 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:16.418295 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-ljm7z" Oct 30 22:34:16.418439 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:16.418324 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-ljm7z" event={"ID":"919d285e-2e4f-41b9-95f8-87144df6a112","Type":"ContainerDied","Data":"6b4e8d5d4fab470b21e36969ed58fe803ba5ccd055d4dd7ef8ceee78bdfe968c"} Oct 30 22:34:16.418439 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:16.418370 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-ljm7z" event={"ID":"919d285e-2e4f-41b9-95f8-87144df6a112","Type":"ContainerDied","Data":"4b60e0ff5e2ab7539479ba3f95edddb1aa044b81c572806f0aa58c619f3850b9"} Oct 30 22:34:16.418439 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:16.418410 6720 scope.go:122] "RemoveContainer" containerID="6b4e8d5d4fab470b21e36969ed58fe803ba5ccd055d4dd7ef8ceee78bdfe968c" Oct 30 22:34:16.428501 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:16.428481 6720 scope.go:122] "RemoveContainer" containerID="6b4e8d5d4fab470b21e36969ed58fe803ba5ccd055d4dd7ef8ceee78bdfe968c" Oct 30 22:34:16.428781 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:34:16.428762 6720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b4e8d5d4fab470b21e36969ed58fe803ba5ccd055d4dd7ef8ceee78bdfe968c\": container with ID starting with 6b4e8d5d4fab470b21e36969ed58fe803ba5ccd055d4dd7ef8ceee78bdfe968c not found: ID does not exist" containerID="6b4e8d5d4fab470b21e36969ed58fe803ba5ccd055d4dd7ef8ceee78bdfe968c" Oct 30 22:34:16.428837 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:16.428790 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b4e8d5d4fab470b21e36969ed58fe803ba5ccd055d4dd7ef8ceee78bdfe968c"} err="failed to get container status \"6b4e8d5d4fab470b21e36969ed58fe803ba5ccd055d4dd7ef8ceee78bdfe968c\": rpc error: code = NotFound desc = could not find container \"6b4e8d5d4fab470b21e36969ed58fe803ba5ccd055d4dd7ef8ceee78bdfe968c\": container with ID starting with 6b4e8d5d4fab470b21e36969ed58fe803ba5ccd055d4dd7ef8ceee78bdfe968c not found: ID does not exist" Oct 30 22:34:16.443708 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:16.443677 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-ljm7z"] Oct 30 22:34:16.448213 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:16.448188 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-1529dc-openshift-default-5d54c4f69-ljm7z"] Oct 30 22:34:17.164990 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:17.164921 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:17.955701 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:17.955665 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="919d285e-2e4f-41b9-95f8-87144df6a112" path="/var/lib/kubelet/pods/919d285e-2e4f-41b9-95f8-87144df6a112/volumes" Oct 30 22:34:18.456398 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:18.456354 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:18.743917 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:18.743839 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:19.462921 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:19.462889 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:19.928973 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:19.928930 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-2-w3-ceb36a-kserve-7c59695467-46gdc" podUID="f1b6e92c-2952-4bcb-ae33-91e7271342fe" containerName="main" containerID="cri-o://0db241e1eb057c733cc8bb28926f2e8cca1a3d419592d08dc7f68a9ebed706d8" gracePeriod=45 Oct 30 22:34:19.947898 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:19.947850 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-ceb36a/premium-w3-ceb36a-kserve-6576bb476f-j5kdr" podUID="d065e788-c972-4683-84ed-86a70ed4791d" containerName="main" containerID="cri-o://01aa584df2fa7f4f1f95b34835fe0ec63e61de670771b7f974407d3033d75cb0" gracePeriod=45 Oct 30 22:34:20.093777 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:20.093733 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-2-w3-ceb36a-kserve-7c59695467-46gdc" podUID="f1b6e92c-2952-4bcb-ae33-91e7271342fe" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": dial tcp 10.134.0.63:8000: connect: connection refused" Oct 30 22:34:20.100963 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:20.100754 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ceb36a/premium-w3-ceb36a-kserve-6576bb476f-j5kdr" podUID="d065e788-c972-4683-84ed-86a70ed4791d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": dial tcp 10.134.0.53:8000: connect: connection refused" Oct 30 22:34:20.149044 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:20.148561 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:20.231735 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:20.231694 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-2-w3-ceb36a-kserve-7c59695467-46gdc" Oct 30 22:34:20.280347 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:20.280315 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ceb36a/premium-w3-ceb36a-kserve-6576bb476f-j5kdr" Oct 30 22:34:20.359288 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:20.359250 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1b6e92c-2952-4bcb-ae33-91e7271342fe-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1b6e92c-2952-4bcb-ae33-91e7271342fe-dshm\") pod \"f1b6e92c-2952-4bcb-ae33-91e7271342fe\" (UID: \"f1b6e92c-2952-4bcb-ae33-91e7271342fe\") " Oct 30 22:34:20.359466 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:20.359324 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d065e788-c972-4683-84ed-86a70ed4791d-kube-api-access-527xz\" (UniqueName: \"kubernetes.io/projected/d065e788-c972-4683-84ed-86a70ed4791d-kube-api-access-527xz\") pod \"d065e788-c972-4683-84ed-86a70ed4791d\" (UID: \"d065e788-c972-4683-84ed-86a70ed4791d\") " Oct 30 22:34:20.359466 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:20.359376 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1b6e92c-2952-4bcb-ae33-91e7271342fe-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1b6e92c-2952-4bcb-ae33-91e7271342fe-model-cache\") pod \"f1b6e92c-2952-4bcb-ae33-91e7271342fe\" (UID: \"f1b6e92c-2952-4bcb-ae33-91e7271342fe\") " Oct 30 22:34:20.359557 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:20.359470 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d065e788-c972-4683-84ed-86a70ed4791d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d065e788-c972-4683-84ed-86a70ed4791d-tmp-dir\") pod \"d065e788-c972-4683-84ed-86a70ed4791d\" (UID: \"d065e788-c972-4683-84ed-86a70ed4791d\") " Oct 30 22:34:20.359557 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:20.359502 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1b6e92c-2952-4bcb-ae33-91e7271342fe-home\" (UniqueName: \"kubernetes.io/empty-dir/f1b6e92c-2952-4bcb-ae33-91e7271342fe-home\") pod \"f1b6e92c-2952-4bcb-ae33-91e7271342fe\" (UID: \"f1b6e92c-2952-4bcb-ae33-91e7271342fe\") " Oct 30 22:34:20.359557 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:20.359548 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d065e788-c972-4683-84ed-86a70ed4791d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d065e788-c972-4683-84ed-86a70ed4791d-dshm\") pod \"d065e788-c972-4683-84ed-86a70ed4791d\" (UID: \"d065e788-c972-4683-84ed-86a70ed4791d\") " Oct 30 22:34:20.359661 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:20.359594 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d065e788-c972-4683-84ed-86a70ed4791d-tls-certs\" (UniqueName: \"kubernetes.io/secret/d065e788-c972-4683-84ed-86a70ed4791d-tls-certs\") pod \"d065e788-c972-4683-84ed-86a70ed4791d\" (UID: \"d065e788-c972-4683-84ed-86a70ed4791d\") " Oct 30 22:34:20.359696 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:20.359659 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1b6e92c-2952-4bcb-ae33-91e7271342fe-tls-certs\" (UniqueName: \"kubernetes.io/secret/f1b6e92c-2952-4bcb-ae33-91e7271342fe-tls-certs\") pod \"f1b6e92c-2952-4bcb-ae33-91e7271342fe\" (UID: \"f1b6e92c-2952-4bcb-ae33-91e7271342fe\") " Oct 30 22:34:20.359731 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:20.359710 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1b6e92c-2952-4bcb-ae33-91e7271342fe-kube-api-access-lqxrv\" (UniqueName: \"kubernetes.io/projected/f1b6e92c-2952-4bcb-ae33-91e7271342fe-kube-api-access-lqxrv\") pod \"f1b6e92c-2952-4bcb-ae33-91e7271342fe\" (UID: \"f1b6e92c-2952-4bcb-ae33-91e7271342fe\") " Oct 30 22:34:20.359772 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:20.359748 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1b6e92c-2952-4bcb-ae33-91e7271342fe-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1b6e92c-2952-4bcb-ae33-91e7271342fe-tmp-dir\") pod \"f1b6e92c-2952-4bcb-ae33-91e7271342fe\" (UID: \"f1b6e92c-2952-4bcb-ae33-91e7271342fe\") " Oct 30 22:34:20.359805 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:20.359794 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d065e788-c972-4683-84ed-86a70ed4791d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d065e788-c972-4683-84ed-86a70ed4791d-model-cache\") pod \"d065e788-c972-4683-84ed-86a70ed4791d\" (UID: \"d065e788-c972-4683-84ed-86a70ed4791d\") " Oct 30 22:34:20.359911 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:20.359893 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d065e788-c972-4683-84ed-86a70ed4791d-home\" (UniqueName: \"kubernetes.io/empty-dir/d065e788-c972-4683-84ed-86a70ed4791d-home\") pod \"d065e788-c972-4683-84ed-86a70ed4791d\" (UID: \"d065e788-c972-4683-84ed-86a70ed4791d\") " Oct 30 22:34:20.360972 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:20.360939 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1b6e92c-2952-4bcb-ae33-91e7271342fe-model-cache" pod "f1b6e92c-2952-4bcb-ae33-91e7271342fe" (UID: "f1b6e92c-2952-4bcb-ae33-91e7271342fe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:20.361166 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:20.361087 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d065e788-c972-4683-84ed-86a70ed4791d-tmp-dir" pod "d065e788-c972-4683-84ed-86a70ed4791d" (UID: "d065e788-c972-4683-84ed-86a70ed4791d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:20.361166 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:20.361155 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1b6e92c-2952-4bcb-ae33-91e7271342fe-tmp-dir" pod "f1b6e92c-2952-4bcb-ae33-91e7271342fe" (UID: "f1b6e92c-2952-4bcb-ae33-91e7271342fe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:20.361330 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:20.361307 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d065e788-c972-4683-84ed-86a70ed4791d-model-cache" pod "d065e788-c972-4683-84ed-86a70ed4791d" (UID: "d065e788-c972-4683-84ed-86a70ed4791d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:20.361554 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:20.361536 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1b6e92c-2952-4bcb-ae33-91e7271342fe-home" pod "f1b6e92c-2952-4bcb-ae33-91e7271342fe" (UID: "f1b6e92c-2952-4bcb-ae33-91e7271342fe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:20.361741 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:20.361720 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d065e788-c972-4683-84ed-86a70ed4791d-home" pod "d065e788-c972-4683-84ed-86a70ed4791d" (UID: "d065e788-c972-4683-84ed-86a70ed4791d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:20.363882 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:20.363853 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1b6e92c-2952-4bcb-ae33-91e7271342fe-kube-api-access-lqxrv" pod "f1b6e92c-2952-4bcb-ae33-91e7271342fe" (UID: "f1b6e92c-2952-4bcb-ae33-91e7271342fe"). InnerVolumeSpecName "kube-api-access-lqxrv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:34:20.364227 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:20.364191 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d065e788-c972-4683-84ed-86a70ed4791d-tls-certs" pod "d065e788-c972-4683-84ed-86a70ed4791d" (UID: "d065e788-c972-4683-84ed-86a70ed4791d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:34:20.364227 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:20.364200 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1b6e92c-2952-4bcb-ae33-91e7271342fe-tls-certs" pod "f1b6e92c-2952-4bcb-ae33-91e7271342fe" (UID: "f1b6e92c-2952-4bcb-ae33-91e7271342fe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:34:20.364489 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:20.364469 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d065e788-c972-4683-84ed-86a70ed4791d-kube-api-access-527xz" pod "d065e788-c972-4683-84ed-86a70ed4791d" (UID: "d065e788-c972-4683-84ed-86a70ed4791d"). InnerVolumeSpecName "kube-api-access-527xz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:34:20.364860 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:20.364832 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d065e788-c972-4683-84ed-86a70ed4791d-dshm" pod "d065e788-c972-4683-84ed-86a70ed4791d" (UID: "d065e788-c972-4683-84ed-86a70ed4791d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:20.365522 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:20.365508 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1b6e92c-2952-4bcb-ae33-91e7271342fe-dshm" pod "f1b6e92c-2952-4bcb-ae33-91e7271342fe" (UID: "f1b6e92c-2952-4bcb-ae33-91e7271342fe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:20.443224 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:20.443192 6720 generic.go:356] "Generic (PLEG): container finished" podID="d065e788-c972-4683-84ed-86a70ed4791d" containerID="01aa584df2fa7f4f1f95b34835fe0ec63e61de670771b7f974407d3033d75cb0" exitCode=0 Oct 30 22:34:20.443471 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:20.443238 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ceb36a/premium-w3-ceb36a-kserve-6576bb476f-j5kdr" Oct 30 22:34:20.443471 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:20.443280 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ceb36a/premium-w3-ceb36a-kserve-6576bb476f-j5kdr" event={"ID":"d065e788-c972-4683-84ed-86a70ed4791d","Type":"ContainerDied","Data":"01aa584df2fa7f4f1f95b34835fe0ec63e61de670771b7f974407d3033d75cb0"} Oct 30 22:34:20.443471 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:20.443322 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ceb36a/premium-w3-ceb36a-kserve-6576bb476f-j5kdr" event={"ID":"d065e788-c972-4683-84ed-86a70ed4791d","Type":"ContainerDied","Data":"cf5bc4a1325d7787b775673abbd83daad39839106a378d0c43fbe6d215764cb6"} Oct 30 22:34:20.443471 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:20.443340 6720 scope.go:122] "RemoveContainer" containerID="01aa584df2fa7f4f1f95b34835fe0ec63e61de670771b7f974407d3033d75cb0" Oct 30 22:34:20.444877 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:20.444851 6720 generic.go:356] "Generic (PLEG): container finished" podID="f1b6e92c-2952-4bcb-ae33-91e7271342fe" containerID="0db241e1eb057c733cc8bb28926f2e8cca1a3d419592d08dc7f68a9ebed706d8" exitCode=0 Oct 30 22:34:20.444968 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:20.444894 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-2-w3-ceb36a-kserve-7c59695467-46gdc" Oct 30 22:34:20.444968 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:20.444913 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-2-w3-ceb36a-kserve-7c59695467-46gdc" event={"ID":"f1b6e92c-2952-4bcb-ae33-91e7271342fe","Type":"ContainerDied","Data":"0db241e1eb057c733cc8bb28926f2e8cca1a3d419592d08dc7f68a9ebed706d8"} Oct 30 22:34:20.445120 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:20.445106 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ceb36a/distinct-2-w3-ceb36a-kserve-7c59695467-46gdc" event={"ID":"f1b6e92c-2952-4bcb-ae33-91e7271342fe","Type":"ContainerDied","Data":"36b31ec83a29ae7f045030a95327dc2936bee3b0d465f67e73dc356e79e1458a"} Oct 30 22:34:20.457993 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:20.457764 6720 scope.go:122] "RemoveContainer" containerID="01aa584df2fa7f4f1f95b34835fe0ec63e61de670771b7f974407d3033d75cb0" Oct 30 22:34:20.458245 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:34:20.458223 6720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01aa584df2fa7f4f1f95b34835fe0ec63e61de670771b7f974407d3033d75cb0\": container with ID starting with 01aa584df2fa7f4f1f95b34835fe0ec63e61de670771b7f974407d3033d75cb0 not found: ID does not exist" containerID="01aa584df2fa7f4f1f95b34835fe0ec63e61de670771b7f974407d3033d75cb0" Oct 30 22:34:20.458343 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:20.458257 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01aa584df2fa7f4f1f95b34835fe0ec63e61de670771b7f974407d3033d75cb0"} err="failed to get container status \"01aa584df2fa7f4f1f95b34835fe0ec63e61de670771b7f974407d3033d75cb0\": rpc error: code = NotFound desc = could not find container \"01aa584df2fa7f4f1f95b34835fe0ec63e61de670771b7f974407d3033d75cb0\": container with ID starting with 01aa584df2fa7f4f1f95b34835fe0ec63e61de670771b7f974407d3033d75cb0 not found: ID does not exist" Oct 30 22:34:20.458343 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:20.458280 6720 scope.go:122] "RemoveContainer" containerID="0db241e1eb057c733cc8bb28926f2e8cca1a3d419592d08dc7f68a9ebed706d8" Oct 30 22:34:20.461882 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:20.461860 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-527xz\" (UniqueName: \"kubernetes.io/projected/d065e788-c972-4683-84ed-86a70ed4791d-kube-api-access-527xz\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:20.461965 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:20.461884 6720 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1b6e92c-2952-4bcb-ae33-91e7271342fe-model-cache\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:20.461965 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:20.461894 6720 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d065e788-c972-4683-84ed-86a70ed4791d-tmp-dir\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:20.461965 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:20.461904 6720 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1b6e92c-2952-4bcb-ae33-91e7271342fe-home\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:20.461965 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:20.461912 6720 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d065e788-c972-4683-84ed-86a70ed4791d-dshm\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:20.461965 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:20.461920 6720 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d065e788-c972-4683-84ed-86a70ed4791d-tls-certs\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:20.461965 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:20.461927 6720 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1b6e92c-2952-4bcb-ae33-91e7271342fe-tls-certs\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:20.461965 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:20.461936 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqxrv\" (UniqueName: \"kubernetes.io/projected/f1b6e92c-2952-4bcb-ae33-91e7271342fe-kube-api-access-lqxrv\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:20.461965 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:20.461945 6720 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1b6e92c-2952-4bcb-ae33-91e7271342fe-tmp-dir\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:20.461965 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:20.461953 6720 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d065e788-c972-4683-84ed-86a70ed4791d-model-cache\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:20.461965 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:20.461961 6720 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d065e788-c972-4683-84ed-86a70ed4791d-home\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:20.461965 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:20.461969 6720 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1b6e92c-2952-4bcb-ae33-91e7271342fe-dshm\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:20.467066 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:20.467048 6720 scope.go:122] "RemoveContainer" containerID="0db241e1eb057c733cc8bb28926f2e8cca1a3d419592d08dc7f68a9ebed706d8" Oct 30 22:34:20.467360 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:34:20.467342 6720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0db241e1eb057c733cc8bb28926f2e8cca1a3d419592d08dc7f68a9ebed706d8\": container with ID starting with 0db241e1eb057c733cc8bb28926f2e8cca1a3d419592d08dc7f68a9ebed706d8 not found: ID does not exist" containerID="0db241e1eb057c733cc8bb28926f2e8cca1a3d419592d08dc7f68a9ebed706d8" Oct 30 22:34:20.467475 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:20.467370 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0db241e1eb057c733cc8bb28926f2e8cca1a3d419592d08dc7f68a9ebed706d8"} err="failed to get container status \"0db241e1eb057c733cc8bb28926f2e8cca1a3d419592d08dc7f68a9ebed706d8\": rpc error: code = NotFound desc = could not find container \"0db241e1eb057c733cc8bb28926f2e8cca1a3d419592d08dc7f68a9ebed706d8\": container with ID starting with 0db241e1eb057c733cc8bb28926f2e8cca1a3d419592d08dc7f68a9ebed706d8 not found: ID does not exist" Oct 30 22:34:20.494590 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:20.494562 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-ceb36a/distinct-2-w3-ceb36a-kserve-7c59695467-46gdc"] Oct 30 22:34:20.501998 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:20.501975 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-ceb36a/distinct-2-w3-ceb36a-kserve-7c59695467-46gdc"] Oct 30 22:34:20.514097 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:20.514074 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-ceb36a/premium-w3-ceb36a-kserve-6576bb476f-j5kdr"] Oct 30 22:34:20.519680 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:20.519656 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-ceb36a/premium-w3-ceb36a-kserve-6576bb476f-j5kdr"] Oct 30 22:34:21.024046 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:21.024017 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:21.754465 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:21.754433 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a39158-65fb5bdfd4-fpwzx"] Oct 30 22:34:21.754978 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:21.754618 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a39158-65fb5bdfd4-fpwzx" podUID="e12f67b4-89de-4975-816d-f44fff4763f9" containerName="maas-api" containerID="cri-o://c7f4811ae2b1f6cd3919febbfe3adbb96f48b56fb12e132a00f67edbb9b5b864" gracePeriod=30 Oct 30 22:34:21.767791 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:21.767752 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:21.963410 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:21.962493 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d065e788-c972-4683-84ed-86a70ed4791d" path="/var/lib/kubelet/pods/d065e788-c972-4683-84ed-86a70ed4791d/volumes" Oct 30 22:34:21.963410 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:21.963099 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1b6e92c-2952-4bcb-ae33-91e7271342fe" path="/var/lib/kubelet/pods/f1b6e92c-2952-4bcb-ae33-91e7271342fe/volumes" Oct 30 22:34:21.997940 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:21.991654 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:22.114300 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:22.114275 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a39158-65fb5bdfd4-fpwzx" Oct 30 22:34:22.256399 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:22.256350 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:22.288881 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:22.288852 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e12f67b4-89de-4975-816d-f44fff4763f9-kube-api-access-msdcw\" (UniqueName: \"kubernetes.io/projected/e12f67b4-89de-4975-816d-f44fff4763f9-kube-api-access-msdcw\") pod \"e12f67b4-89de-4975-816d-f44fff4763f9\" (UID: \"e12f67b4-89de-4975-816d-f44fff4763f9\") " Oct 30 22:34:22.289033 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:22.288892 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e12f67b4-89de-4975-816d-f44fff4763f9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e12f67b4-89de-4975-816d-f44fff4763f9-openshift-service-ca\") pod \"e12f67b4-89de-4975-816d-f44fff4763f9\" (UID: \"e12f67b4-89de-4975-816d-f44fff4763f9\") " Oct 30 22:34:22.289033 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:22.288941 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e12f67b4-89de-4975-816d-f44fff4763f9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e12f67b4-89de-4975-816d-f44fff4763f9-metrics-certs\") pod \"e12f67b4-89de-4975-816d-f44fff4763f9\" (UID: \"e12f67b4-89de-4975-816d-f44fff4763f9\") " Oct 30 22:34:22.289033 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:22.288983 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e12f67b4-89de-4975-816d-f44fff4763f9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e12f67b4-89de-4975-816d-f44fff4763f9-maas-api-tls\") pod \"e12f67b4-89de-4975-816d-f44fff4763f9\" (UID: \"e12f67b4-89de-4975-816d-f44fff4763f9\") " Oct 30 22:34:22.289368 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:22.289342 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e12f67b4-89de-4975-816d-f44fff4763f9-openshift-service-ca" pod "e12f67b4-89de-4975-816d-f44fff4763f9" (UID: "e12f67b4-89de-4975-816d-f44fff4763f9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:34:22.291274 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:22.291245 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e12f67b4-89de-4975-816d-f44fff4763f9-metrics-certs" pod "e12f67b4-89de-4975-816d-f44fff4763f9" (UID: "e12f67b4-89de-4975-816d-f44fff4763f9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:34:22.291274 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:22.291261 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e12f67b4-89de-4975-816d-f44fff4763f9-maas-api-tls" pod "e12f67b4-89de-4975-816d-f44fff4763f9" (UID: "e12f67b4-89de-4975-816d-f44fff4763f9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:34:22.291445 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:22.291283 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e12f67b4-89de-4975-816d-f44fff4763f9-kube-api-access-msdcw" pod "e12f67b4-89de-4975-816d-f44fff4763f9" (UID: "e12f67b4-89de-4975-816d-f44fff4763f9"). InnerVolumeSpecName "kube-api-access-msdcw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:34:22.390338 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:22.390299 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msdcw\" (UniqueName: \"kubernetes.io/projected/e12f67b4-89de-4975-816d-f44fff4763f9-kube-api-access-msdcw\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:22.390338 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:22.390334 6720 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e12f67b4-89de-4975-816d-f44fff4763f9-openshift-service-ca\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:22.390338 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:22.390347 6720 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e12f67b4-89de-4975-816d-f44fff4763f9-metrics-certs\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:22.390674 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:22.390356 6720 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e12f67b4-89de-4975-816d-f44fff4763f9-maas-api-tls\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:22.444378 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:22.444344 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:22.462175 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:22.462131 6720 generic.go:356] "Generic (PLEG): container finished" podID="e12f67b4-89de-4975-816d-f44fff4763f9" containerID="c7f4811ae2b1f6cd3919febbfe3adbb96f48b56fb12e132a00f67edbb9b5b864" exitCode=0 Oct 30 22:34:22.462400 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:22.462182 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a39158-65fb5bdfd4-fpwzx" Oct 30 22:34:22.462400 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:22.462211 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a39158-65fb5bdfd4-fpwzx" event={"ID":"e12f67b4-89de-4975-816d-f44fff4763f9","Type":"ContainerDied","Data":"c7f4811ae2b1f6cd3919febbfe3adbb96f48b56fb12e132a00f67edbb9b5b864"} Oct 30 22:34:22.462400 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:22.462284 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-a39158-65fb5bdfd4-fpwzx" event={"ID":"e12f67b4-89de-4975-816d-f44fff4763f9","Type":"ContainerDied","Data":"3f163065b7169e2c004fb22540abfb304e11c4488aae2dacfdb97d077aa8bbf1"} Oct 30 22:34:22.462400 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:22.462317 6720 scope.go:122] "RemoveContainer" containerID="c7f4811ae2b1f6cd3919febbfe3adbb96f48b56fb12e132a00f67edbb9b5b864" Oct 30 22:34:22.475196 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:22.475174 6720 scope.go:122] "RemoveContainer" containerID="c7f4811ae2b1f6cd3919febbfe3adbb96f48b56fb12e132a00f67edbb9b5b864" Oct 30 22:34:22.475539 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:34:22.475518 6720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7f4811ae2b1f6cd3919febbfe3adbb96f48b56fb12e132a00f67edbb9b5b864\": container with ID starting with c7f4811ae2b1f6cd3919febbfe3adbb96f48b56fb12e132a00f67edbb9b5b864 not found: ID does not exist" containerID="c7f4811ae2b1f6cd3919febbfe3adbb96f48b56fb12e132a00f67edbb9b5b864" Oct 30 22:34:22.475640 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:22.475545 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7f4811ae2b1f6cd3919febbfe3adbb96f48b56fb12e132a00f67edbb9b5b864"} err="failed to get container status \"c7f4811ae2b1f6cd3919febbfe3adbb96f48b56fb12e132a00f67edbb9b5b864\": rpc error: code = NotFound desc = could not find container \"c7f4811ae2b1f6cd3919febbfe3adbb96f48b56fb12e132a00f67edbb9b5b864\": container with ID starting with c7f4811ae2b1f6cd3919febbfe3adbb96f48b56fb12e132a00f67edbb9b5b864 not found: ID does not exist" Oct 30 22:34:22.497325 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:22.496212 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a39158-65fb5bdfd4-fpwzx"] Oct 30 22:34:22.510908 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:22.510878 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-a39158-65fb5bdfd4-fpwzx"] Oct 30 22:34:22.810870 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:22.810659 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:23.569356 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:23.569314 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:23.960557 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:23.960519 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e12f67b4-89de-4975-816d-f44fff4763f9" path="/var/lib/kubelet/pods/e12f67b4-89de-4975-816d-f44fff4763f9/volumes" Oct 30 22:34:24.647999 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:24.647961 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:25.581137 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:25.581101 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:26.237096 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:26.237062 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-ceb36a-7b9fbcd9d5-kssjr"] Oct 30 22:34:26.237358 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:26.237328 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ceb36a-7b9fbcd9d5-kssjr" podUID="22c4c993-7bb7-492e-a4bc-05e7fa6c7b7f" containerName="maas-api" containerID="cri-o://60998a3d9c680ea9ddaa6eaf35d03a44b7e335890afcbc71657943bb7f92cba0" gracePeriod=30 Oct 30 22:34:26.379746 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:26.379005 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:26.494414 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:26.494293 6720 generic.go:356] "Generic (PLEG): container finished" podID="22c4c993-7bb7-492e-a4bc-05e7fa6c7b7f" containerID="60998a3d9c680ea9ddaa6eaf35d03a44b7e335890afcbc71657943bb7f92cba0" exitCode=0 Oct 30 22:34:26.494414 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:26.494367 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ceb36a-7b9fbcd9d5-kssjr" event={"ID":"22c4c993-7bb7-492e-a4bc-05e7fa6c7b7f","Type":"ContainerDied","Data":"60998a3d9c680ea9ddaa6eaf35d03a44b7e335890afcbc71657943bb7f92cba0"} Oct 30 22:34:26.536818 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:26.536794 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ceb36a-7b9fbcd9d5-kssjr" Oct 30 22:34:26.637237 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:26.637206 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22c4c993-7bb7-492e-a4bc-05e7fa6c7b7f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22c4c993-7bb7-492e-a4bc-05e7fa6c7b7f-openshift-service-ca\") pod \"22c4c993-7bb7-492e-a4bc-05e7fa6c7b7f\" (UID: \"22c4c993-7bb7-492e-a4bc-05e7fa6c7b7f\") " Oct 30 22:34:26.637237 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:26.637243 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22c4c993-7bb7-492e-a4bc-05e7fa6c7b7f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/22c4c993-7bb7-492e-a4bc-05e7fa6c7b7f-metrics-certs\") pod \"22c4c993-7bb7-492e-a4bc-05e7fa6c7b7f\" (UID: \"22c4c993-7bb7-492e-a4bc-05e7fa6c7b7f\") " Oct 30 22:34:26.637794 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:26.637268 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22c4c993-7bb7-492e-a4bc-05e7fa6c7b7f-kube-api-access-dj2l6\" (UniqueName: \"kubernetes.io/projected/22c4c993-7bb7-492e-a4bc-05e7fa6c7b7f-kube-api-access-dj2l6\") pod \"22c4c993-7bb7-492e-a4bc-05e7fa6c7b7f\" (UID: \"22c4c993-7bb7-492e-a4bc-05e7fa6c7b7f\") " Oct 30 22:34:26.637794 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:26.637374 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22c4c993-7bb7-492e-a4bc-05e7fa6c7b7f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22c4c993-7bb7-492e-a4bc-05e7fa6c7b7f-maas-api-tls\") pod \"22c4c993-7bb7-492e-a4bc-05e7fa6c7b7f\" (UID: \"22c4c993-7bb7-492e-a4bc-05e7fa6c7b7f\") " Oct 30 22:34:26.637794 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:26.637615 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22c4c993-7bb7-492e-a4bc-05e7fa6c7b7f-openshift-service-ca" pod "22c4c993-7bb7-492e-a4bc-05e7fa6c7b7f" (UID: "22c4c993-7bb7-492e-a4bc-05e7fa6c7b7f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:34:26.637911 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:26.637819 6720 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22c4c993-7bb7-492e-a4bc-05e7fa6c7b7f-openshift-service-ca\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:26.639620 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:26.639596 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22c4c993-7bb7-492e-a4bc-05e7fa6c7b7f-maas-api-tls" pod "22c4c993-7bb7-492e-a4bc-05e7fa6c7b7f" (UID: "22c4c993-7bb7-492e-a4bc-05e7fa6c7b7f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:34:26.640051 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:26.640028 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22c4c993-7bb7-492e-a4bc-05e7fa6c7b7f-metrics-certs" pod "22c4c993-7bb7-492e-a4bc-05e7fa6c7b7f" (UID: "22c4c993-7bb7-492e-a4bc-05e7fa6c7b7f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:34:26.640140 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:26.640074 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22c4c993-7bb7-492e-a4bc-05e7fa6c7b7f-kube-api-access-dj2l6" pod "22c4c993-7bb7-492e-a4bc-05e7fa6c7b7f" (UID: "22c4c993-7bb7-492e-a4bc-05e7fa6c7b7f"). InnerVolumeSpecName "kube-api-access-dj2l6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:34:26.692995 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:26.692960 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:26.738847 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:26.738803 6720 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22c4c993-7bb7-492e-a4bc-05e7fa6c7b7f-maas-api-tls\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:26.738847 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:26.738838 6720 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22c4c993-7bb7-492e-a4bc-05e7fa6c7b7f-metrics-certs\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:26.738847 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:26.738852 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dj2l6\" (UniqueName: \"kubernetes.io/projected/22c4c993-7bb7-492e-a4bc-05e7fa6c7b7f-kube-api-access-dj2l6\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:26.968563 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:26.968531 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:27.214241 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:27.214207 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:27.441669 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:27.441639 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:27.501930 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:27.501900 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ceb36a-7b9fbcd9d5-kssjr" Oct 30 22:34:27.502131 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:27.501931 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ceb36a-7b9fbcd9d5-kssjr" event={"ID":"22c4c993-7bb7-492e-a4bc-05e7fa6c7b7f","Type":"ContainerDied","Data":"34b151fbcd28be599c2bb2f838e96c06708c86aadcde66ff5778ad07b52155f0"} Oct 30 22:34:27.502131 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:27.501997 6720 scope.go:122] "RemoveContainer" containerID="60998a3d9c680ea9ddaa6eaf35d03a44b7e335890afcbc71657943bb7f92cba0" Oct 30 22:34:27.531367 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:27.531339 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-ceb36a-7b9fbcd9d5-kssjr"] Oct 30 22:34:27.536527 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:27.536506 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-ceb36a-7b9fbcd9d5-kssjr"] Oct 30 22:34:27.895682 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:27.895651 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:27.955867 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:27.955830 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22c4c993-7bb7-492e-a4bc-05e7fa6c7b7f" path="/var/lib/kubelet/pods/22c4c993-7bb7-492e-a4bc-05e7fa6c7b7f/volumes" Oct 30 22:34:28.785361 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:28.785328 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:29.631343 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:29.631298 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:30.318098 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:30.318045 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:30.574035 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:30.573939 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:30.778959 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:30.778903 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:30.991171 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:30.991142 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:31.332499 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:31.332411 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:32.348641 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:32.348603 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:33.161378 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:33.161342 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:33.819254 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:33.817635 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:34.466449 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:34.466410 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:34.691202 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:34.691172 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:34.898629 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:34.898595 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:35.060061 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:35.060026 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:35.188861 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:35.188781 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-rvzw9"] Oct 30 22:34:35.189273 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:35.189243 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-rvzw9" podUID="cc1b52ec-a38c-409e-810c-7ae3998c2ebd" containerName="istio-proxy" containerID="cri-o://4c32c1e5c36e5394efc19774aceeb4b04f6086c5347f34532bbe4bd3d99edafe" gracePeriod=30 Oct 30 22:34:35.258069 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:35.258037 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:35.540458 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:35.534575 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:35.731345 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:35.731269 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:36.325781 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:36.325742 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:37.125884 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:37.125832 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:37.836199 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:37.836137 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:38.520833 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:38.520797 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:39.363376 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:39.363343 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:40.024149 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:40.024118 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:40.413690 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:34:40.413651 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 22:34:40.448752 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:40.448732 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-rvzw9" Oct 30 22:34:40.476232 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:40.476199 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc1b52ec-a38c-409e-810c-7ae3998c2ebd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc1b52ec-a38c-409e-810c-7ae3998c2ebd-istio-envoy\") pod \"cc1b52ec-a38c-409e-810c-7ae3998c2ebd\" (UID: \"cc1b52ec-a38c-409e-810c-7ae3998c2ebd\") " Oct 30 22:34:40.476432 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:40.476246 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc1b52ec-a38c-409e-810c-7ae3998c2ebd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc1b52ec-a38c-409e-810c-7ae3998c2ebd-istiod-ca-cert\") pod \"cc1b52ec-a38c-409e-810c-7ae3998c2ebd\" (UID: \"cc1b52ec-a38c-409e-810c-7ae3998c2ebd\") " Oct 30 22:34:40.476432 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:40.476319 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc1b52ec-a38c-409e-810c-7ae3998c2ebd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc1b52ec-a38c-409e-810c-7ae3998c2ebd-credential-socket\") pod \"cc1b52ec-a38c-409e-810c-7ae3998c2ebd\" (UID: \"cc1b52ec-a38c-409e-810c-7ae3998c2ebd\") " Oct 30 22:34:40.476432 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:40.476371 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc1b52ec-a38c-409e-810c-7ae3998c2ebd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc1b52ec-a38c-409e-810c-7ae3998c2ebd-workload-socket\") pod \"cc1b52ec-a38c-409e-810c-7ae3998c2ebd\" (UID: \"cc1b52ec-a38c-409e-810c-7ae3998c2ebd\") " Oct 30 22:34:40.476702 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:40.476435 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc1b52ec-a38c-409e-810c-7ae3998c2ebd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc1b52ec-a38c-409e-810c-7ae3998c2ebd-istio-data\") pod \"cc1b52ec-a38c-409e-810c-7ae3998c2ebd\" (UID: \"cc1b52ec-a38c-409e-810c-7ae3998c2ebd\") " Oct 30 22:34:40.476702 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:40.476479 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc1b52ec-a38c-409e-810c-7ae3998c2ebd-kube-api-access-5vlpg\" (UniqueName: \"kubernetes.io/projected/cc1b52ec-a38c-409e-810c-7ae3998c2ebd-kube-api-access-5vlpg\") pod \"cc1b52ec-a38c-409e-810c-7ae3998c2ebd\" (UID: \"cc1b52ec-a38c-409e-810c-7ae3998c2ebd\") " Oct 30 22:34:40.476702 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:40.476543 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc1b52ec-a38c-409e-810c-7ae3998c2ebd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc1b52ec-a38c-409e-810c-7ae3998c2ebd-workload-certs\") pod \"cc1b52ec-a38c-409e-810c-7ae3998c2ebd\" (UID: \"cc1b52ec-a38c-409e-810c-7ae3998c2ebd\") " Oct 30 22:34:40.476702 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:40.476590 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cc1b52ec-a38c-409e-810c-7ae3998c2ebd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc1b52ec-a38c-409e-810c-7ae3998c2ebd-istio-podinfo\") pod \"cc1b52ec-a38c-409e-810c-7ae3998c2ebd\" (UID: \"cc1b52ec-a38c-409e-810c-7ae3998c2ebd\") " Oct 30 22:34:40.476702 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:40.476631 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc1b52ec-a38c-409e-810c-7ae3998c2ebd-istio-token\" (UniqueName: \"kubernetes.io/projected/cc1b52ec-a38c-409e-810c-7ae3998c2ebd-istio-token\") pod \"cc1b52ec-a38c-409e-810c-7ae3998c2ebd\" (UID: \"cc1b52ec-a38c-409e-810c-7ae3998c2ebd\") " Oct 30 22:34:40.476702 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:40.476686 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc1b52ec-a38c-409e-810c-7ae3998c2ebd-credential-socket" pod "cc1b52ec-a38c-409e-810c-7ae3998c2ebd" (UID: "cc1b52ec-a38c-409e-810c-7ae3998c2ebd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:40.477020 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:40.476697 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc1b52ec-a38c-409e-810c-7ae3998c2ebd-workload-socket" pod "cc1b52ec-a38c-409e-810c-7ae3998c2ebd" (UID: "cc1b52ec-a38c-409e-810c-7ae3998c2ebd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:40.477020 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:40.476769 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc1b52ec-a38c-409e-810c-7ae3998c2ebd-istiod-ca-cert" pod "cc1b52ec-a38c-409e-810c-7ae3998c2ebd" (UID: "cc1b52ec-a38c-409e-810c-7ae3998c2ebd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:34:40.477020 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:40.476934 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc1b52ec-a38c-409e-810c-7ae3998c2ebd-workload-certs" pod "cc1b52ec-a38c-409e-810c-7ae3998c2ebd" (UID: "cc1b52ec-a38c-409e-810c-7ae3998c2ebd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:40.477352 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:40.477291 6720 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc1b52ec-a38c-409e-810c-7ae3998c2ebd-istiod-ca-cert\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:40.477352 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:40.477321 6720 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc1b52ec-a38c-409e-810c-7ae3998c2ebd-credential-socket\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:40.477352 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:40.477336 6720 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc1b52ec-a38c-409e-810c-7ae3998c2ebd-workload-socket\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:40.477352 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:40.477350 6720 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc1b52ec-a38c-409e-810c-7ae3998c2ebd-workload-certs\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:40.477869 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:40.477848 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc1b52ec-a38c-409e-810c-7ae3998c2ebd-istio-data" pod "cc1b52ec-a38c-409e-810c-7ae3998c2ebd" (UID: "cc1b52ec-a38c-409e-810c-7ae3998c2ebd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:40.479639 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:40.479598 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc1b52ec-a38c-409e-810c-7ae3998c2ebd-istio-token" pod "cc1b52ec-a38c-409e-810c-7ae3998c2ebd" (UID: "cc1b52ec-a38c-409e-810c-7ae3998c2ebd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:34:40.479748 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:40.479721 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc1b52ec-a38c-409e-810c-7ae3998c2ebd-kube-api-access-5vlpg" pod "cc1b52ec-a38c-409e-810c-7ae3998c2ebd" (UID: "cc1b52ec-a38c-409e-810c-7ae3998c2ebd"). InnerVolumeSpecName "kube-api-access-5vlpg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:34:40.479870 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:40.479812 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc1b52ec-a38c-409e-810c-7ae3998c2ebd-istio-envoy" pod "cc1b52ec-a38c-409e-810c-7ae3998c2ebd" (UID: "cc1b52ec-a38c-409e-810c-7ae3998c2ebd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:40.480208 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:40.480182 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc1b52ec-a38c-409e-810c-7ae3998c2ebd-istio-podinfo" pod "cc1b52ec-a38c-409e-810c-7ae3998c2ebd" (UID: "cc1b52ec-a38c-409e-810c-7ae3998c2ebd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:34:40.578067 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:40.577968 6720 generic.go:356] "Generic (PLEG): container finished" podID="cc1b52ec-a38c-409e-810c-7ae3998c2ebd" containerID="4c32c1e5c36e5394efc19774aceeb4b04f6086c5347f34532bbe4bd3d99edafe" exitCode=0 Oct 30 22:34:40.578067 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:40.578055 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-rvzw9" Oct 30 22:34:40.578312 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:40.578084 6720 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc1b52ec-a38c-409e-810c-7ae3998c2ebd-istio-podinfo\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:40.578312 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:40.578085 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-rvzw9" event={"ID":"cc1b52ec-a38c-409e-810c-7ae3998c2ebd","Type":"ContainerDied","Data":"4c32c1e5c36e5394efc19774aceeb4b04f6086c5347f34532bbe4bd3d99edafe"} Oct 30 22:34:40.578312 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:40.578102 6720 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc1b52ec-a38c-409e-810c-7ae3998c2ebd-istio-token\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:40.578312 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:40.578116 6720 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc1b52ec-a38c-409e-810c-7ae3998c2ebd-istio-envoy\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:40.578312 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:40.578128 6720 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc1b52ec-a38c-409e-810c-7ae3998c2ebd-istio-data\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:40.578312 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:40.578141 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vlpg\" (UniqueName: \"kubernetes.io/projected/cc1b52ec-a38c-409e-810c-7ae3998c2ebd-kube-api-access-5vlpg\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:40.578312 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:40.578143 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-rvzw9" event={"ID":"cc1b52ec-a38c-409e-810c-7ae3998c2ebd","Type":"ContainerDied","Data":"f947ff9093e901903eb19ede176ad88e4b67b1815b6569d8bf006986692ebec8"} Oct 30 22:34:40.578312 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:40.578171 6720 scope.go:122] "RemoveContainer" containerID="4c32c1e5c36e5394efc19774aceeb4b04f6086c5347f34532bbe4bd3d99edafe" Oct 30 22:34:40.589201 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:40.589178 6720 scope.go:122] "RemoveContainer" containerID="4c32c1e5c36e5394efc19774aceeb4b04f6086c5347f34532bbe4bd3d99edafe" Oct 30 22:34:40.589468 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:34:40.589449 6720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c32c1e5c36e5394efc19774aceeb4b04f6086c5347f34532bbe4bd3d99edafe\": container with ID starting with 4c32c1e5c36e5394efc19774aceeb4b04f6086c5347f34532bbe4bd3d99edafe not found: ID does not exist" containerID="4c32c1e5c36e5394efc19774aceeb4b04f6086c5347f34532bbe4bd3d99edafe" Oct 30 22:34:40.589571 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:40.589480 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c32c1e5c36e5394efc19774aceeb4b04f6086c5347f34532bbe4bd3d99edafe"} err="failed to get container status \"4c32c1e5c36e5394efc19774aceeb4b04f6086c5347f34532bbe4bd3d99edafe\": rpc error: code = NotFound desc = could not find container \"4c32c1e5c36e5394efc19774aceeb4b04f6086c5347f34532bbe4bd3d99edafe\": container with ID starting with 4c32c1e5c36e5394efc19774aceeb4b04f6086c5347f34532bbe4bd3d99edafe not found: ID does not exist" Oct 30 22:34:40.602390 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:40.602366 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-rvzw9"] Oct 30 22:34:40.609860 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:40.609835 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-ceb36a-openshift-default-866d6d8fbc-rvzw9"] Oct 30 22:34:40.800584 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:40.800509 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:41.531597 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:41.531550 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:41.679648 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:41.679615 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:41.839716 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:41.839615 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:41.956301 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:41.956268 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc1b52ec-a38c-409e-810c-7ae3998c2ebd" path="/var/lib/kubelet/pods/cc1b52ec-a38c-409e-810c-7ae3998c2ebd/volumes" Oct 30 22:34:42.038317 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:42.038281 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-26a128-846fd4d8c6-z4mcb"] Oct 30 22:34:42.038564 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:42.038532 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-26a128-846fd4d8c6-z4mcb" podUID="9c5ce4b5-7597-47c4-9530-56544816a191" containerName="maas-api" containerID="cri-o://b2cdbbbd6e47e3342bc72ff84d140aca08fe5e40e3cffc12e4da7635c07c50cf" gracePeriod=30 Oct 30 22:34:42.145115 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:42.145087 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:42.296189 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:42.296145 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:42.354970 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:42.354939 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-26a128-846fd4d8c6-z4mcb" Oct 30 22:34:42.396269 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:42.396186 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c5ce4b5-7597-47c4-9530-56544816a191-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c5ce4b5-7597-47c4-9530-56544816a191-openshift-service-ca\") pod \"9c5ce4b5-7597-47c4-9530-56544816a191\" (UID: \"9c5ce4b5-7597-47c4-9530-56544816a191\") " Oct 30 22:34:42.396483 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:42.396275 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c5ce4b5-7597-47c4-9530-56544816a191-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c5ce4b5-7597-47c4-9530-56544816a191-maas-api-tls\") pod \"9c5ce4b5-7597-47c4-9530-56544816a191\" (UID: \"9c5ce4b5-7597-47c4-9530-56544816a191\") " Oct 30 22:34:42.396483 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:42.396328 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c5ce4b5-7597-47c4-9530-56544816a191-kube-api-access-bwlpx\" (UniqueName: \"kubernetes.io/projected/9c5ce4b5-7597-47c4-9530-56544816a191-kube-api-access-bwlpx\") pod \"9c5ce4b5-7597-47c4-9530-56544816a191\" (UID: \"9c5ce4b5-7597-47c4-9530-56544816a191\") " Oct 30 22:34:42.396483 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:42.396364 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c5ce4b5-7597-47c4-9530-56544816a191-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c5ce4b5-7597-47c4-9530-56544816a191-metrics-certs\") pod \"9c5ce4b5-7597-47c4-9530-56544816a191\" (UID: \"9c5ce4b5-7597-47c4-9530-56544816a191\") " Oct 30 22:34:42.396681 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:42.396652 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c5ce4b5-7597-47c4-9530-56544816a191-openshift-service-ca" pod "9c5ce4b5-7597-47c4-9530-56544816a191" (UID: "9c5ce4b5-7597-47c4-9530-56544816a191"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:34:42.396866 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:42.396838 6720 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c5ce4b5-7597-47c4-9530-56544816a191-openshift-service-ca\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:42.400961 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:42.400924 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c5ce4b5-7597-47c4-9530-56544816a191-maas-api-tls" pod "9c5ce4b5-7597-47c4-9530-56544816a191" (UID: "9c5ce4b5-7597-47c4-9530-56544816a191"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:34:42.401087 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:42.400926 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c5ce4b5-7597-47c4-9530-56544816a191-kube-api-access-bwlpx" pod "9c5ce4b5-7597-47c4-9530-56544816a191" (UID: "9c5ce4b5-7597-47c4-9530-56544816a191"). InnerVolumeSpecName "kube-api-access-bwlpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:34:42.401087 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:42.401023 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c5ce4b5-7597-47c4-9530-56544816a191-metrics-certs" pod "9c5ce4b5-7597-47c4-9530-56544816a191" (UID: "9c5ce4b5-7597-47c4-9530-56544816a191"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:34:42.460352 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:42.460319 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:42.497749 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:42.497712 6720 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c5ce4b5-7597-47c4-9530-56544816a191-maas-api-tls\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:42.497749 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:42.497744 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwlpx\" (UniqueName: \"kubernetes.io/projected/9c5ce4b5-7597-47c4-9530-56544816a191-kube-api-access-bwlpx\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:42.497749 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:42.497757 6720 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c5ce4b5-7597-47c4-9530-56544816a191-metrics-certs\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:42.595029 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:42.594996 6720 generic.go:356] "Generic (PLEG): container finished" podID="9c5ce4b5-7597-47c4-9530-56544816a191" containerID="b2cdbbbd6e47e3342bc72ff84d140aca08fe5e40e3cffc12e4da7635c07c50cf" exitCode=0 Oct 30 22:34:42.595544 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:42.595036 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-26a128-846fd4d8c6-z4mcb" event={"ID":"9c5ce4b5-7597-47c4-9530-56544816a191","Type":"ContainerDied","Data":"b2cdbbbd6e47e3342bc72ff84d140aca08fe5e40e3cffc12e4da7635c07c50cf"} Oct 30 22:34:42.595544 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:42.595049 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-26a128-846fd4d8c6-z4mcb" Oct 30 22:34:42.595544 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:42.595079 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-26a128-846fd4d8c6-z4mcb" event={"ID":"9c5ce4b5-7597-47c4-9530-56544816a191","Type":"ContainerDied","Data":"e9a2972280b52ff309ff17490d665131e1b0e40c79edbda69294a34d967568af"} Oct 30 22:34:42.595544 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:42.595102 6720 scope.go:122] "RemoveContainer" containerID="b2cdbbbd6e47e3342bc72ff84d140aca08fe5e40e3cffc12e4da7635c07c50cf" Oct 30 22:34:42.605753 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:42.605727 6720 scope.go:122] "RemoveContainer" containerID="b2cdbbbd6e47e3342bc72ff84d140aca08fe5e40e3cffc12e4da7635c07c50cf" Oct 30 22:34:42.606031 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:34:42.606007 6720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2cdbbbd6e47e3342bc72ff84d140aca08fe5e40e3cffc12e4da7635c07c50cf\": container with ID starting with b2cdbbbd6e47e3342bc72ff84d140aca08fe5e40e3cffc12e4da7635c07c50cf not found: ID does not exist" containerID="b2cdbbbd6e47e3342bc72ff84d140aca08fe5e40e3cffc12e4da7635c07c50cf" Oct 30 22:34:42.606094 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:42.606042 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2cdbbbd6e47e3342bc72ff84d140aca08fe5e40e3cffc12e4da7635c07c50cf"} err="failed to get container status \"b2cdbbbd6e47e3342bc72ff84d140aca08fe5e40e3cffc12e4da7635c07c50cf\": rpc error: code = NotFound desc = could not find container \"b2cdbbbd6e47e3342bc72ff84d140aca08fe5e40e3cffc12e4da7635c07c50cf\": container with ID starting with b2cdbbbd6e47e3342bc72ff84d140aca08fe5e40e3cffc12e4da7635c07c50cf not found: ID does not exist" Oct 30 22:34:42.642840 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:42.642807 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-26a128-846fd4d8c6-z4mcb"] Oct 30 22:34:42.650831 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:42.650747 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-26a128-846fd4d8c6-z4mcb"] Oct 30 22:34:42.752257 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:42.752224 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:43.048577 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:43.048500 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:43.541172 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:43.541040 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:43.695038 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:43.694969 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:43.956479 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:43.956447 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c5ce4b5-7597-47c4-9530-56544816a191" path="/var/lib/kubelet/pods/9c5ce4b5-7597-47c4-9530-56544816a191/volumes" Oct 30 22:34:44.444855 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:44.444566 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:44.704005 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:44.703905 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:44.878335 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:44.878296 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:45.473083 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:45.473037 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:46.457713 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:46.457671 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:47.392496 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:47.392454 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:48.237404 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:48.237347 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:49.222839 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:49.222805 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:49.922506 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:49.922459 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:50.483890 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:50.483848 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:50.629119 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:50.628878 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:50.815936 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:50.815818 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:50.969150 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:50.969096 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:51.525344 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:51.525305 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:52.201993 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:52.201960 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:52.349163 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:52.349121 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:52.927093 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:52.927057 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:53.607275 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:53.607231 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:54.224993 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:54.224956 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:54.900946 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:54.900910 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:55.634277 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:55.634243 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:56.375304 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:56.375248 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:57.032132 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:57.032100 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:57.725708 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:57.725660 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:58.431047 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:58.430991 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:59.120316 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:59.120278 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:34:59.722242 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:34:59.722177 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:35:00.410615 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:00.410571 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:35:01.117590 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:01.117561 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:35:01.665789 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:01.665749 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:35:01.819331 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:01.819289 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:35:02.107254 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:02.107208 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:35:02.259648 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:02.259609 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:35:02.689084 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:02.689042 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:35:02.844062 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:02.844024 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:35:03.582472 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:03.582436 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:35:04.325499 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:04.325465 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:35:05.011925 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:05.011890 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:35:05.214200 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:05.214160 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:35:05.360635 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:05.360603 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:35:05.950248 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:05.950218 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:35:06.610363 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:06.608316 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:35:07.343868 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:07.343837 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:35:08.001194 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:08.001162 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:35:08.678034 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:08.677989 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:35:09.515148 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:09.515034 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:35:09.667378 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:09.667336 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:35:09.923096 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:09.923067 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:35:10.406512 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:35:10.406463 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 22:35:10.524986 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:10.524959 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:35:11.231852 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:11.231821 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:35:12.031093 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:12.031051 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:35:12.751774 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:12.751714 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:35:13.426916 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:13.426883 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:35:14.165621 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:14.165589 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:35:14.949489 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:14.949445 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:35:15.838019 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:15.837968 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:35:16.693336 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:16.693304 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:35:17.417632 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:17.417580 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:35:18.125865 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:18.125827 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:35:18.916792 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:18.916613 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:35:19.709358 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:19.709324 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:35:20.324895 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:20.324856 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:35:20.732038 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:20.732007 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:35:20.883116 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:20.883038 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:35:21.042583 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:21.042499 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:35:21.461923 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:21.461886 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-38d6e2-868dddd9c8-cv8wc"] Oct 30 22:35:21.462243 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:21.462215 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-38d6e2-868dddd9c8-cv8wc" podUID="883a9007-3704-47d0-b59e-8591e1d33186" containerName="maas-api" containerID="cri-o://f637de80f4276a149cf7718d95893d6e4cc9059d6c6f957167d69d4fd2171c0e" gracePeriod=30 Oct 30 22:35:21.562601 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:21.562565 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:35:21.724162 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:21.724083 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:35:21.750744 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:21.750706 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-38d6e2-868dddd9c8-cv8wc" Oct 30 22:35:21.806453 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:21.806416 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/883a9007-3704-47d0-b59e-8591e1d33186-kube-api-access-fzgcg\" (UniqueName: \"kubernetes.io/projected/883a9007-3704-47d0-b59e-8591e1d33186-kube-api-access-fzgcg\") pod \"883a9007-3704-47d0-b59e-8591e1d33186\" (UID: \"883a9007-3704-47d0-b59e-8591e1d33186\") " Oct 30 22:35:21.806673 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:21.806515 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/883a9007-3704-47d0-b59e-8591e1d33186-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/883a9007-3704-47d0-b59e-8591e1d33186-openshift-service-ca\") pod \"883a9007-3704-47d0-b59e-8591e1d33186\" (UID: \"883a9007-3704-47d0-b59e-8591e1d33186\") " Oct 30 22:35:21.806673 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:21.806567 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/883a9007-3704-47d0-b59e-8591e1d33186-metrics-certs\" (UniqueName: \"kubernetes.io/secret/883a9007-3704-47d0-b59e-8591e1d33186-metrics-certs\") pod \"883a9007-3704-47d0-b59e-8591e1d33186\" (UID: \"883a9007-3704-47d0-b59e-8591e1d33186\") " Oct 30 22:35:21.806673 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:21.806657 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/883a9007-3704-47d0-b59e-8591e1d33186-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/883a9007-3704-47d0-b59e-8591e1d33186-maas-api-tls\") pod \"883a9007-3704-47d0-b59e-8591e1d33186\" (UID: \"883a9007-3704-47d0-b59e-8591e1d33186\") " Oct 30 22:35:21.806935 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:21.806911 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/883a9007-3704-47d0-b59e-8591e1d33186-openshift-service-ca" pod "883a9007-3704-47d0-b59e-8591e1d33186" (UID: "883a9007-3704-47d0-b59e-8591e1d33186"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:35:21.809222 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:21.809179 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/883a9007-3704-47d0-b59e-8591e1d33186-kube-api-access-fzgcg" pod "883a9007-3704-47d0-b59e-8591e1d33186" (UID: "883a9007-3704-47d0-b59e-8591e1d33186"). InnerVolumeSpecName "kube-api-access-fzgcg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:35:21.809345 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:21.809256 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/883a9007-3704-47d0-b59e-8591e1d33186-metrics-certs" pod "883a9007-3704-47d0-b59e-8591e1d33186" (UID: "883a9007-3704-47d0-b59e-8591e1d33186"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:35:21.809427 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:21.809371 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/883a9007-3704-47d0-b59e-8591e1d33186-maas-api-tls" pod "883a9007-3704-47d0-b59e-8591e1d33186" (UID: "883a9007-3704-47d0-b59e-8591e1d33186"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:35:21.816577 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:21.816551 6720 generic.go:356] "Generic (PLEG): container finished" podID="883a9007-3704-47d0-b59e-8591e1d33186" containerID="f637de80f4276a149cf7718d95893d6e4cc9059d6c6f957167d69d4fd2171c0e" exitCode=0 Oct 30 22:35:21.816695 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:21.816599 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-38d6e2-868dddd9c8-cv8wc" Oct 30 22:35:21.816695 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:21.816633 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-38d6e2-868dddd9c8-cv8wc" event={"ID":"883a9007-3704-47d0-b59e-8591e1d33186","Type":"ContainerDied","Data":"f637de80f4276a149cf7718d95893d6e4cc9059d6c6f957167d69d4fd2171c0e"} Oct 30 22:35:21.816695 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:21.816686 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-38d6e2-868dddd9c8-cv8wc" event={"ID":"883a9007-3704-47d0-b59e-8591e1d33186","Type":"ContainerDied","Data":"d23f15d79da237aa7b378c1d4c95c472d12ce2f56bf6f96035b6fcc0cdd0234e"} Oct 30 22:35:21.816803 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:21.816707 6720 scope.go:122] "RemoveContainer" containerID="f637de80f4276a149cf7718d95893d6e4cc9059d6c6f957167d69d4fd2171c0e" Oct 30 22:35:21.834277 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:21.834252 6720 scope.go:122] "RemoveContainer" containerID="f637de80f4276a149cf7718d95893d6e4cc9059d6c6f957167d69d4fd2171c0e" Oct 30 22:35:21.834588 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:35:21.834568 6720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f637de80f4276a149cf7718d95893d6e4cc9059d6c6f957167d69d4fd2171c0e\": container with ID starting with f637de80f4276a149cf7718d95893d6e4cc9059d6c6f957167d69d4fd2171c0e not found: ID does not exist" containerID="f637de80f4276a149cf7718d95893d6e4cc9059d6c6f957167d69d4fd2171c0e" Oct 30 22:35:21.834669 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:21.834600 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f637de80f4276a149cf7718d95893d6e4cc9059d6c6f957167d69d4fd2171c0e"} err="failed to get container status \"f637de80f4276a149cf7718d95893d6e4cc9059d6c6f957167d69d4fd2171c0e\": rpc error: code = NotFound desc = could not find container \"f637de80f4276a149cf7718d95893d6e4cc9059d6c6f957167d69d4fd2171c0e\": container with ID starting with f637de80f4276a149cf7718d95893d6e4cc9059d6c6f957167d69d4fd2171c0e not found: ID does not exist" Oct 30 22:35:21.847248 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:21.847220 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-38d6e2-868dddd9c8-cv8wc"] Oct 30 22:35:21.856959 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:21.856936 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-38d6e2-868dddd9c8-cv8wc"] Oct 30 22:35:21.879569 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:21.879497 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:35:21.907848 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:21.907822 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzgcg\" (UniqueName: \"kubernetes.io/projected/883a9007-3704-47d0-b59e-8591e1d33186-kube-api-access-fzgcg\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:35:21.907848 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:21.907846 6720 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/883a9007-3704-47d0-b59e-8591e1d33186-openshift-service-ca\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:35:21.908027 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:21.907861 6720 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/883a9007-3704-47d0-b59e-8591e1d33186-metrics-certs\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:35:21.908027 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:21.907870 6720 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/883a9007-3704-47d0-b59e-8591e1d33186-maas-api-tls\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:35:21.957040 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:21.957001 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="883a9007-3704-47d0-b59e-8591e1d33186" path="/var/lib/kubelet/pods/883a9007-3704-47d0-b59e-8591e1d33186/volumes" Oct 30 22:35:22.025854 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:22.025770 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:35:22.381175 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:22.381139 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:35:22.708037 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:22.707954 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:35:23.464403 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:23.464356 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:35:24.229926 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:24.229888 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:35:24.926771 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:24.926726 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:35:25.676642 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:25.676608 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:35:26.339677 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:26.339646 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:35:27.028846 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:27.028809 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:35:27.737266 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:27.737229 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:35:28.438005 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:28.437972 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:35:29.089214 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:29.089178 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:35:29.235068 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:29.235032 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:35:29.462077 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:29.462043 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:35:29.830092 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:29.829952 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:35:30.002615 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:30.002583 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:35:30.588800 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:30.588760 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:35:30.819215 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:30.819188 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:35:31.516346 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:31.516296 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:35:32.549058 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:32.549027 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:35:33.226031 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:33.225978 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:35:33.985935 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:33.985906 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:35:34.859691 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:34.859653 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:35:34.958368 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:34.958331 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-939a16/distinct-w4-939a16-kserve-7f88f68c56-cngtk"] Oct 30 22:35:34.995841 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:34.995806 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-939a16/simulator-w4-939a16-kserve-74f845bf84-4c7b6"] Oct 30 22:35:35.144441 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:35.144334 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:35:35.759709 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:35.759634 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:35:36.428561 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:36.428513 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:35:37.233532 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:37.233494 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:35:40.403488 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:35:40.403444 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 22:35:49.979474 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:49.979413 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-939a16/distinct-w4-939a16-kserve-7f88f68c56-cngtk" podUID="85316c69-7ebd-4fb6-9195-44d1fba2dfbc" containerName="main" containerID="cri-o://171820a5d8cdf5c032aecaaa7c783a831940885f2d5065de1c2159b3ae8adad2" gracePeriod=45 Oct 30 22:35:50.017585 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:50.017544 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-939a16/simulator-w4-939a16-kserve-74f845bf84-4c7b6" podUID="9bb6805f-f355-4ea8-80b7-542215c42022" containerName="main" containerID="cri-o://fb9912a8eb18f42b39c59f14b1b4c1ea131db15d2f7569520a8eb46d6cc51fde" gracePeriod=45 Oct 30 22:35:50.294991 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:50.294965 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-939a16/distinct-w4-939a16-kserve-7f88f68c56-cngtk" Oct 30 22:35:50.312088 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:50.312064 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-939a16/simulator-w4-939a16-kserve-74f845bf84-4c7b6" Oct 30 22:35:50.315544 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:50.315517 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85316c69-7ebd-4fb6-9195-44d1fba2dfbc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85316c69-7ebd-4fb6-9195-44d1fba2dfbc-model-cache\") pod \"85316c69-7ebd-4fb6-9195-44d1fba2dfbc\" (UID: \"85316c69-7ebd-4fb6-9195-44d1fba2dfbc\") " Oct 30 22:35:50.315702 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:50.315575 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85316c69-7ebd-4fb6-9195-44d1fba2dfbc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/85316c69-7ebd-4fb6-9195-44d1fba2dfbc-dshm\") pod \"85316c69-7ebd-4fb6-9195-44d1fba2dfbc\" (UID: \"85316c69-7ebd-4fb6-9195-44d1fba2dfbc\") " Oct 30 22:35:50.315702 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:50.315611 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85316c69-7ebd-4fb6-9195-44d1fba2dfbc-tls-certs\" (UniqueName: \"kubernetes.io/secret/85316c69-7ebd-4fb6-9195-44d1fba2dfbc-tls-certs\") pod \"85316c69-7ebd-4fb6-9195-44d1fba2dfbc\" (UID: \"85316c69-7ebd-4fb6-9195-44d1fba2dfbc\") " Oct 30 22:35:50.315702 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:50.315637 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85316c69-7ebd-4fb6-9195-44d1fba2dfbc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85316c69-7ebd-4fb6-9195-44d1fba2dfbc-tmp-dir\") pod \"85316c69-7ebd-4fb6-9195-44d1fba2dfbc\" (UID: \"85316c69-7ebd-4fb6-9195-44d1fba2dfbc\") " Oct 30 22:35:50.316151 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:50.316090 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85316c69-7ebd-4fb6-9195-44d1fba2dfbc-kube-api-access-hlhzx\" (UniqueName: \"kubernetes.io/projected/85316c69-7ebd-4fb6-9195-44d1fba2dfbc-kube-api-access-hlhzx\") pod \"85316c69-7ebd-4fb6-9195-44d1fba2dfbc\" (UID: \"85316c69-7ebd-4fb6-9195-44d1fba2dfbc\") " Oct 30 22:35:50.316331 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:50.316300 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85316c69-7ebd-4fb6-9195-44d1fba2dfbc-home\" (UniqueName: \"kubernetes.io/empty-dir/85316c69-7ebd-4fb6-9195-44d1fba2dfbc-home\") pod \"85316c69-7ebd-4fb6-9195-44d1fba2dfbc\" (UID: \"85316c69-7ebd-4fb6-9195-44d1fba2dfbc\") " Oct 30 22:35:50.317017 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:50.316992 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85316c69-7ebd-4fb6-9195-44d1fba2dfbc-home" pod "85316c69-7ebd-4fb6-9195-44d1fba2dfbc" (UID: "85316c69-7ebd-4fb6-9195-44d1fba2dfbc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:35:50.317464 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:50.317425 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85316c69-7ebd-4fb6-9195-44d1fba2dfbc-model-cache" pod "85316c69-7ebd-4fb6-9195-44d1fba2dfbc" (UID: "85316c69-7ebd-4fb6-9195-44d1fba2dfbc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:35:50.317804 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:50.317766 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85316c69-7ebd-4fb6-9195-44d1fba2dfbc-tmp-dir" pod "85316c69-7ebd-4fb6-9195-44d1fba2dfbc" (UID: "85316c69-7ebd-4fb6-9195-44d1fba2dfbc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:35:50.320590 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:50.320559 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85316c69-7ebd-4fb6-9195-44d1fba2dfbc-tls-certs" pod "85316c69-7ebd-4fb6-9195-44d1fba2dfbc" (UID: "85316c69-7ebd-4fb6-9195-44d1fba2dfbc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:35:50.320696 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:50.320643 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85316c69-7ebd-4fb6-9195-44d1fba2dfbc-dshm" pod "85316c69-7ebd-4fb6-9195-44d1fba2dfbc" (UID: "85316c69-7ebd-4fb6-9195-44d1fba2dfbc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:35:50.321021 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:50.320990 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85316c69-7ebd-4fb6-9195-44d1fba2dfbc-kube-api-access-hlhzx" pod "85316c69-7ebd-4fb6-9195-44d1fba2dfbc" (UID: "85316c69-7ebd-4fb6-9195-44d1fba2dfbc"). InnerVolumeSpecName "kube-api-access-hlhzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:35:50.417887 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:50.417423 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bb6805f-f355-4ea8-80b7-542215c42022-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bb6805f-f355-4ea8-80b7-542215c42022-tmp-dir\") pod \"9bb6805f-f355-4ea8-80b7-542215c42022\" (UID: \"9bb6805f-f355-4ea8-80b7-542215c42022\") " Oct 30 22:35:50.417887 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:50.417477 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bb6805f-f355-4ea8-80b7-542215c42022-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bb6805f-f355-4ea8-80b7-542215c42022-model-cache\") pod \"9bb6805f-f355-4ea8-80b7-542215c42022\" (UID: \"9bb6805f-f355-4ea8-80b7-542215c42022\") " Oct 30 22:35:50.417887 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:50.417537 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9bb6805f-f355-4ea8-80b7-542215c42022-tls-certs\" (UniqueName: \"kubernetes.io/secret/9bb6805f-f355-4ea8-80b7-542215c42022-tls-certs\") pod \"9bb6805f-f355-4ea8-80b7-542215c42022\" (UID: \"9bb6805f-f355-4ea8-80b7-542215c42022\") " Oct 30 22:35:50.417887 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:50.417567 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bb6805f-f355-4ea8-80b7-542215c42022-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bb6805f-f355-4ea8-80b7-542215c42022-dshm\") pod \"9bb6805f-f355-4ea8-80b7-542215c42022\" (UID: \"9bb6805f-f355-4ea8-80b7-542215c42022\") " Oct 30 22:35:50.417887 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:50.417683 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bb6805f-f355-4ea8-80b7-542215c42022-home\" (UniqueName: \"kubernetes.io/empty-dir/9bb6805f-f355-4ea8-80b7-542215c42022-home\") pod \"9bb6805f-f355-4ea8-80b7-542215c42022\" (UID: \"9bb6805f-f355-4ea8-80b7-542215c42022\") " Oct 30 22:35:50.417887 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:50.417770 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bb6805f-f355-4ea8-80b7-542215c42022-kube-api-access-zrxwm\" (UniqueName: \"kubernetes.io/projected/9bb6805f-f355-4ea8-80b7-542215c42022-kube-api-access-zrxwm\") pod \"9bb6805f-f355-4ea8-80b7-542215c42022\" (UID: \"9bb6805f-f355-4ea8-80b7-542215c42022\") " Oct 30 22:35:50.419094 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:50.418333 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bb6805f-f355-4ea8-80b7-542215c42022-model-cache" pod "9bb6805f-f355-4ea8-80b7-542215c42022" (UID: "9bb6805f-f355-4ea8-80b7-542215c42022"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:35:50.419094 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:50.418443 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bb6805f-f355-4ea8-80b7-542215c42022-home" pod "9bb6805f-f355-4ea8-80b7-542215c42022" (UID: "9bb6805f-f355-4ea8-80b7-542215c42022"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:35:50.419279 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:50.419103 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bb6805f-f355-4ea8-80b7-542215c42022-tmp-dir" pod "9bb6805f-f355-4ea8-80b7-542215c42022" (UID: "9bb6805f-f355-4ea8-80b7-542215c42022"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:35:50.419279 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:50.419158 6720 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85316c69-7ebd-4fb6-9195-44d1fba2dfbc-tls-certs\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:35:50.419279 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:50.419183 6720 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85316c69-7ebd-4fb6-9195-44d1fba2dfbc-tmp-dir\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:35:50.419279 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:50.419199 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlhzx\" (UniqueName: \"kubernetes.io/projected/85316c69-7ebd-4fb6-9195-44d1fba2dfbc-kube-api-access-hlhzx\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:35:50.419279 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:50.419214 6720 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bb6805f-f355-4ea8-80b7-542215c42022-home\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:35:50.419279 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:50.419229 6720 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85316c69-7ebd-4fb6-9195-44d1fba2dfbc-home\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:35:50.419279 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:50.419243 6720 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bb6805f-f355-4ea8-80b7-542215c42022-model-cache\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:35:50.419279 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:50.419257 6720 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85316c69-7ebd-4fb6-9195-44d1fba2dfbc-model-cache\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:35:50.419279 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:50.419272 6720 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85316c69-7ebd-4fb6-9195-44d1fba2dfbc-dshm\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:35:50.420370 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:50.420343 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bb6805f-f355-4ea8-80b7-542215c42022-dshm" pod "9bb6805f-f355-4ea8-80b7-542215c42022" (UID: "9bb6805f-f355-4ea8-80b7-542215c42022"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:35:50.420521 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:50.420467 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bb6805f-f355-4ea8-80b7-542215c42022-tls-certs" pod "9bb6805f-f355-4ea8-80b7-542215c42022" (UID: "9bb6805f-f355-4ea8-80b7-542215c42022"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:35:50.420566 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:50.420518 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bb6805f-f355-4ea8-80b7-542215c42022-kube-api-access-zrxwm" pod "9bb6805f-f355-4ea8-80b7-542215c42022" (UID: "9bb6805f-f355-4ea8-80b7-542215c42022"). InnerVolumeSpecName "kube-api-access-zrxwm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:35:50.520306 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:50.520225 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrxwm\" (UniqueName: \"kubernetes.io/projected/9bb6805f-f355-4ea8-80b7-542215c42022-kube-api-access-zrxwm\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:35:50.520306 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:50.520256 6720 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bb6805f-f355-4ea8-80b7-542215c42022-tmp-dir\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:35:50.520306 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:50.520265 6720 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bb6805f-f355-4ea8-80b7-542215c42022-tls-certs\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:35:50.520306 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:50.520274 6720 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bb6805f-f355-4ea8-80b7-542215c42022-dshm\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:35:50.970493 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:50.970454 6720 generic.go:356] "Generic (PLEG): container finished" podID="9bb6805f-f355-4ea8-80b7-542215c42022" containerID="fb9912a8eb18f42b39c59f14b1b4c1ea131db15d2f7569520a8eb46d6cc51fde" exitCode=0 Oct 30 22:35:50.970715 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:50.970508 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-939a16/simulator-w4-939a16-kserve-74f845bf84-4c7b6" Oct 30 22:35:50.970715 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:50.970542 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-939a16/simulator-w4-939a16-kserve-74f845bf84-4c7b6" event={"ID":"9bb6805f-f355-4ea8-80b7-542215c42022","Type":"ContainerDied","Data":"fb9912a8eb18f42b39c59f14b1b4c1ea131db15d2f7569520a8eb46d6cc51fde"} Oct 30 22:35:50.970715 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:50.970593 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-939a16/simulator-w4-939a16-kserve-74f845bf84-4c7b6" event={"ID":"9bb6805f-f355-4ea8-80b7-542215c42022","Type":"ContainerDied","Data":"6a6b15e4d09a0b7fac5ccdd62c8c069a531b25fdf5e70df69472d061d534c4b4"} Oct 30 22:35:50.970715 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:50.970615 6720 scope.go:122] "RemoveContainer" containerID="fb9912a8eb18f42b39c59f14b1b4c1ea131db15d2f7569520a8eb46d6cc51fde" Oct 30 22:35:50.972043 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:50.972019 6720 generic.go:356] "Generic (PLEG): container finished" podID="85316c69-7ebd-4fb6-9195-44d1fba2dfbc" containerID="171820a5d8cdf5c032aecaaa7c783a831940885f2d5065de1c2159b3ae8adad2" exitCode=0 Oct 30 22:35:50.972168 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:50.972065 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-939a16/distinct-w4-939a16-kserve-7f88f68c56-cngtk" Oct 30 22:35:50.972168 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:50.972077 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-939a16/distinct-w4-939a16-kserve-7f88f68c56-cngtk" event={"ID":"85316c69-7ebd-4fb6-9195-44d1fba2dfbc","Type":"ContainerDied","Data":"171820a5d8cdf5c032aecaaa7c783a831940885f2d5065de1c2159b3ae8adad2"} Oct 30 22:35:50.972168 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:50.972109 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-939a16/distinct-w4-939a16-kserve-7f88f68c56-cngtk" event={"ID":"85316c69-7ebd-4fb6-9195-44d1fba2dfbc","Type":"ContainerDied","Data":"5f9a70dd029eacdb2af4836cddeabe1e24a5f218937d677e0d3e68a5261989de"} Oct 30 22:35:50.981515 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:50.981325 6720 scope.go:122] "RemoveContainer" containerID="fb9912a8eb18f42b39c59f14b1b4c1ea131db15d2f7569520a8eb46d6cc51fde" Oct 30 22:35:50.981782 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:35:50.981629 6720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb9912a8eb18f42b39c59f14b1b4c1ea131db15d2f7569520a8eb46d6cc51fde\": container with ID starting with fb9912a8eb18f42b39c59f14b1b4c1ea131db15d2f7569520a8eb46d6cc51fde not found: ID does not exist" containerID="fb9912a8eb18f42b39c59f14b1b4c1ea131db15d2f7569520a8eb46d6cc51fde" Oct 30 22:35:50.981782 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:50.981668 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb9912a8eb18f42b39c59f14b1b4c1ea131db15d2f7569520a8eb46d6cc51fde"} err="failed to get container status \"fb9912a8eb18f42b39c59f14b1b4c1ea131db15d2f7569520a8eb46d6cc51fde\": rpc error: code = NotFound desc = could not find container \"fb9912a8eb18f42b39c59f14b1b4c1ea131db15d2f7569520a8eb46d6cc51fde\": container with ID starting with fb9912a8eb18f42b39c59f14b1b4c1ea131db15d2f7569520a8eb46d6cc51fde not found: ID does not exist" Oct 30 22:35:50.981782 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:50.981691 6720 scope.go:122] "RemoveContainer" containerID="171820a5d8cdf5c032aecaaa7c783a831940885f2d5065de1c2159b3ae8adad2" Oct 30 22:35:50.990810 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:50.990789 6720 scope.go:122] "RemoveContainer" containerID="171820a5d8cdf5c032aecaaa7c783a831940885f2d5065de1c2159b3ae8adad2" Oct 30 22:35:50.991087 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:35:50.991063 6720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"171820a5d8cdf5c032aecaaa7c783a831940885f2d5065de1c2159b3ae8adad2\": container with ID starting with 171820a5d8cdf5c032aecaaa7c783a831940885f2d5065de1c2159b3ae8adad2 not found: ID does not exist" containerID="171820a5d8cdf5c032aecaaa7c783a831940885f2d5065de1c2159b3ae8adad2" Oct 30 22:35:50.991185 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:50.991090 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"171820a5d8cdf5c032aecaaa7c783a831940885f2d5065de1c2159b3ae8adad2"} err="failed to get container status \"171820a5d8cdf5c032aecaaa7c783a831940885f2d5065de1c2159b3ae8adad2\": rpc error: code = NotFound desc = could not find container \"171820a5d8cdf5c032aecaaa7c783a831940885f2d5065de1c2159b3ae8adad2\": container with ID starting with 171820a5d8cdf5c032aecaaa7c783a831940885f2d5065de1c2159b3ae8adad2 not found: ID does not exist" Oct 30 22:35:50.993599 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:50.993577 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-939a16/simulator-w4-939a16-kserve-74f845bf84-4c7b6"] Oct 30 22:35:51.000157 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:51.000137 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-939a16/simulator-w4-939a16-kserve-74f845bf84-4c7b6"] Oct 30 22:35:51.012360 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:51.012328 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-939a16/distinct-w4-939a16-kserve-7f88f68c56-cngtk"] Oct 30 22:35:51.016188 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:51.016163 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-939a16/distinct-w4-939a16-kserve-7f88f68c56-cngtk"] Oct 30 22:35:51.955617 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:51.955587 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85316c69-7ebd-4fb6-9195-44d1fba2dfbc" path="/var/lib/kubelet/pods/85316c69-7ebd-4fb6-9195-44d1fba2dfbc/volumes" Oct 30 22:35:51.956024 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:51.956011 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bb6805f-f355-4ea8-80b7-542215c42022" path="/var/lib/kubelet/pods/9bb6805f-f355-4ea8-80b7-542215c42022/volumes" Oct 30 22:35:52.766957 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:52.766927 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:35:52.866633 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:52.866605 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:35:55.321909 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:55.321872 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-939a16-86877564f-dvkvf"] Oct 30 22:35:55.322568 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:55.322088 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-939a16-86877564f-dvkvf" podUID="2f501cad-6ca4-4615-b978-bd05ee0cfa2c" containerName="maas-api" containerID="cri-o://dfe42498929435064cb8601420efdad80c099517a0d209b74158bbc26b9dbdc0" gracePeriod=30 Oct 30 22:35:55.419796 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:55.419759 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:35:55.580824 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:55.580742 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:35:55.615207 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:55.615180 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-939a16-86877564f-dvkvf" Oct 30 22:35:55.674175 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:55.674137 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f501cad-6ca4-4615-b978-bd05ee0cfa2c-kube-api-access-spd9f\" (UniqueName: \"kubernetes.io/projected/2f501cad-6ca4-4615-b978-bd05ee0cfa2c-kube-api-access-spd9f\") pod \"2f501cad-6ca4-4615-b978-bd05ee0cfa2c\" (UID: \"2f501cad-6ca4-4615-b978-bd05ee0cfa2c\") " Oct 30 22:35:55.674375 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:55.674188 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f501cad-6ca4-4615-b978-bd05ee0cfa2c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f501cad-6ca4-4615-b978-bd05ee0cfa2c-openshift-service-ca\") pod \"2f501cad-6ca4-4615-b978-bd05ee0cfa2c\" (UID: \"2f501cad-6ca4-4615-b978-bd05ee0cfa2c\") " Oct 30 22:35:55.674375 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:55.674325 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f501cad-6ca4-4615-b978-bd05ee0cfa2c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f501cad-6ca4-4615-b978-bd05ee0cfa2c-metrics-certs\") pod \"2f501cad-6ca4-4615-b978-bd05ee0cfa2c\" (UID: \"2f501cad-6ca4-4615-b978-bd05ee0cfa2c\") " Oct 30 22:35:55.674375 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:55.674360 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f501cad-6ca4-4615-b978-bd05ee0cfa2c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f501cad-6ca4-4615-b978-bd05ee0cfa2c-maas-api-tls\") pod \"2f501cad-6ca4-4615-b978-bd05ee0cfa2c\" (UID: \"2f501cad-6ca4-4615-b978-bd05ee0cfa2c\") " Oct 30 22:35:55.674640 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:55.674602 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f501cad-6ca4-4615-b978-bd05ee0cfa2c-openshift-service-ca" pod "2f501cad-6ca4-4615-b978-bd05ee0cfa2c" (UID: "2f501cad-6ca4-4615-b978-bd05ee0cfa2c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:35:55.674819 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:55.674801 6720 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f501cad-6ca4-4615-b978-bd05ee0cfa2c-openshift-service-ca\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:35:55.676689 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:55.676664 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f501cad-6ca4-4615-b978-bd05ee0cfa2c-metrics-certs" pod "2f501cad-6ca4-4615-b978-bd05ee0cfa2c" (UID: "2f501cad-6ca4-4615-b978-bd05ee0cfa2c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:35:55.676781 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:55.676690 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f501cad-6ca4-4615-b978-bd05ee0cfa2c-kube-api-access-spd9f" pod "2f501cad-6ca4-4615-b978-bd05ee0cfa2c" (UID: "2f501cad-6ca4-4615-b978-bd05ee0cfa2c"). InnerVolumeSpecName "kube-api-access-spd9f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:35:55.676781 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:55.676694 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f501cad-6ca4-4615-b978-bd05ee0cfa2c-maas-api-tls" pod "2f501cad-6ca4-4615-b978-bd05ee0cfa2c" (UID: "2f501cad-6ca4-4615-b978-bd05ee0cfa2c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:35:55.729506 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:55.729475 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:35:55.775611 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:55.775576 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spd9f\" (UniqueName: \"kubernetes.io/projected/2f501cad-6ca4-4615-b978-bd05ee0cfa2c-kube-api-access-spd9f\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:35:55.775611 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:55.775603 6720 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f501cad-6ca4-4615-b978-bd05ee0cfa2c-metrics-certs\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:35:55.775611 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:55.775612 6720 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f501cad-6ca4-4615-b978-bd05ee0cfa2c-maas-api-tls\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:35:56.000593 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:56.000559 6720 generic.go:356] "Generic (PLEG): container finished" podID="2f501cad-6ca4-4615-b978-bd05ee0cfa2c" containerID="dfe42498929435064cb8601420efdad80c099517a0d209b74158bbc26b9dbdc0" exitCode=0 Oct 30 22:35:56.000812 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:56.000613 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-939a16-86877564f-dvkvf" Oct 30 22:35:56.000812 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:56.000650 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-939a16-86877564f-dvkvf" event={"ID":"2f501cad-6ca4-4615-b978-bd05ee0cfa2c","Type":"ContainerDied","Data":"dfe42498929435064cb8601420efdad80c099517a0d209b74158bbc26b9dbdc0"} Oct 30 22:35:56.000812 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:56.000712 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-939a16-86877564f-dvkvf" event={"ID":"2f501cad-6ca4-4615-b978-bd05ee0cfa2c","Type":"ContainerDied","Data":"31694c2dc677fd39d0d046571e86ae8d6367f0bb7200f843c69c37fbf7d8c35e"} Oct 30 22:35:56.000812 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:56.000741 6720 scope.go:122] "RemoveContainer" containerID="dfe42498929435064cb8601420efdad80c099517a0d209b74158bbc26b9dbdc0" Oct 30 22:35:56.010194 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:56.010174 6720 scope.go:122] "RemoveContainer" containerID="dfe42498929435064cb8601420efdad80c099517a0d209b74158bbc26b9dbdc0" Oct 30 22:35:56.010476 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:35:56.010456 6720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfe42498929435064cb8601420efdad80c099517a0d209b74158bbc26b9dbdc0\": container with ID starting with dfe42498929435064cb8601420efdad80c099517a0d209b74158bbc26b9dbdc0 not found: ID does not exist" containerID="dfe42498929435064cb8601420efdad80c099517a0d209b74158bbc26b9dbdc0" Oct 30 22:35:56.010528 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:56.010486 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfe42498929435064cb8601420efdad80c099517a0d209b74158bbc26b9dbdc0"} err="failed to get container status \"dfe42498929435064cb8601420efdad80c099517a0d209b74158bbc26b9dbdc0\": rpc error: code = NotFound desc = could not find container \"dfe42498929435064cb8601420efdad80c099517a0d209b74158bbc26b9dbdc0\": container with ID starting with dfe42498929435064cb8601420efdad80c099517a0d209b74158bbc26b9dbdc0 not found: ID does not exist" Oct 30 22:35:56.019180 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:56.019156 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-939a16-86877564f-dvkvf"] Oct 30 22:35:56.023625 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:56.023601 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-939a16-86877564f-dvkvf"] Oct 30 22:35:56.159032 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:56.159000 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:35:57.955909 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:35:57.955879 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f501cad-6ca4-4615-b978-bd05ee0cfa2c" path="/var/lib/kubelet/pods/2f501cad-6ca4-4615-b978-bd05ee0cfa2c/volumes" Oct 30 22:36:02.023547 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:02.023425 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:36:02.171482 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:02.171447 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:36:02.304673 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:02.304587 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-nv8nz"] Oct 30 22:36:02.305137 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:02.305104 6720 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-nv8nz" podUID="852dbcb8-3c2d-4622-beeb-3fc265a7f906" containerName="istio-proxy" containerID="cri-o://dffc857ed3bc39215ce2a2834d0d81edb8861fe1f13dc1b926ef70b2246d4ae5" gracePeriod=30 Oct 30 22:36:02.399256 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:02.399194 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:36:03.341484 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:03.341449 6720 patch_prober.go:29] interesting pod/e2e-worker-w4-939a16-openshift-default-5fccf878b7-nv8nz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:36:03.341942 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:03.341502 6720 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-nv8nz" podUID="852dbcb8-3c2d-4622-beeb-3fc265a7f906" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:36:03.476234 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:03.476205 6720 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-87c457c84-q9js5_bd0b4436-451d-4abd-aba0-3ebb841ba48e/authorino/0.log" Oct 30 22:36:04.878686 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:04.878656 6720 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-x5rxl_1bb7e9b5-60a0-4cf9-a5ab-2e5e76d077aa/manager/0.log" Oct 30 22:36:05.860202 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:05.860170 6720 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-cptxw_9a04fd45-c758-4a59-8c04-c9499862dcbd/manager/0.log" Oct 30 22:36:06.267331 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:06.267238 6720 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-phjxr_753e9e9a-5c68-4540-8fc7-858514ca428e/manager/0.log" Oct 30 22:36:06.762651 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:06.762615 6720 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-57d6c9d545-ntxm7_91b3aaaf-cf84-44d0-9156-7174e41f8bed/manager/0.log" Oct 30 22:36:06.950781 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:06.950753 6720 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-7c7vd_0cefd95b-2c8c-4271-93b3-4d37e6e00d8e/server/0.log" Oct 30 22:36:07.142557 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:07.142523 6720 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-ckx8d_838cabb6-566e-4376-8c49-f0801ac171fc/manager/0.log" Oct 30 22:36:07.557678 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:07.557651 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-nv8nz" Oct 30 22:36:07.698579 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:07.698493 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/852dbcb8-3c2d-4622-beeb-3fc265a7f906-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/852dbcb8-3c2d-4622-beeb-3fc265a7f906-istiod-ca-cert\") pod \"852dbcb8-3c2d-4622-beeb-3fc265a7f906\" (UID: \"852dbcb8-3c2d-4622-beeb-3fc265a7f906\") " Oct 30 22:36:07.698579 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:07.698555 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/852dbcb8-3c2d-4622-beeb-3fc265a7f906-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/852dbcb8-3c2d-4622-beeb-3fc265a7f906-credential-socket\") pod \"852dbcb8-3c2d-4622-beeb-3fc265a7f906\" (UID: \"852dbcb8-3c2d-4622-beeb-3fc265a7f906\") " Oct 30 22:36:07.698824 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:07.698598 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/852dbcb8-3c2d-4622-beeb-3fc265a7f906-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/852dbcb8-3c2d-4622-beeb-3fc265a7f906-workload-certs\") pod \"852dbcb8-3c2d-4622-beeb-3fc265a7f906\" (UID: \"852dbcb8-3c2d-4622-beeb-3fc265a7f906\") " Oct 30 22:36:07.698824 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:07.698635 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/852dbcb8-3c2d-4622-beeb-3fc265a7f906-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/852dbcb8-3c2d-4622-beeb-3fc265a7f906-workload-socket\") pod \"852dbcb8-3c2d-4622-beeb-3fc265a7f906\" (UID: \"852dbcb8-3c2d-4622-beeb-3fc265a7f906\") " Oct 30 22:36:07.698824 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:07.698678 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/852dbcb8-3c2d-4622-beeb-3fc265a7f906-kube-api-access-v2lf4\" (UniqueName: \"kubernetes.io/projected/852dbcb8-3c2d-4622-beeb-3fc265a7f906-kube-api-access-v2lf4\") pod \"852dbcb8-3c2d-4622-beeb-3fc265a7f906\" (UID: \"852dbcb8-3c2d-4622-beeb-3fc265a7f906\") " Oct 30 22:36:07.698824 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:07.698709 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/852dbcb8-3c2d-4622-beeb-3fc265a7f906-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/852dbcb8-3c2d-4622-beeb-3fc265a7f906-istio-envoy\") pod \"852dbcb8-3c2d-4622-beeb-3fc265a7f906\" (UID: \"852dbcb8-3c2d-4622-beeb-3fc265a7f906\") " Oct 30 22:36:07.698824 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:07.698782 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/852dbcb8-3c2d-4622-beeb-3fc265a7f906-istio-token\" (UniqueName: \"kubernetes.io/projected/852dbcb8-3c2d-4622-beeb-3fc265a7f906-istio-token\") pod \"852dbcb8-3c2d-4622-beeb-3fc265a7f906\" (UID: \"852dbcb8-3c2d-4622-beeb-3fc265a7f906\") " Oct 30 22:36:07.698824 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:07.698809 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/852dbcb8-3c2d-4622-beeb-3fc265a7f906-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/852dbcb8-3c2d-4622-beeb-3fc265a7f906-istio-podinfo\") pod \"852dbcb8-3c2d-4622-beeb-3fc265a7f906\" (UID: \"852dbcb8-3c2d-4622-beeb-3fc265a7f906\") " Oct 30 22:36:07.699140 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:07.698875 6720 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/852dbcb8-3c2d-4622-beeb-3fc265a7f906-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/852dbcb8-3c2d-4622-beeb-3fc265a7f906-istio-data\") pod \"852dbcb8-3c2d-4622-beeb-3fc265a7f906\" (UID: \"852dbcb8-3c2d-4622-beeb-3fc265a7f906\") " Oct 30 22:36:07.699140 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:07.698948 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/852dbcb8-3c2d-4622-beeb-3fc265a7f906-credential-socket" pod "852dbcb8-3c2d-4622-beeb-3fc265a7f906" (UID: "852dbcb8-3c2d-4622-beeb-3fc265a7f906"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:07.699140 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:07.698967 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/852dbcb8-3c2d-4622-beeb-3fc265a7f906-workload-certs" pod "852dbcb8-3c2d-4622-beeb-3fc265a7f906" (UID: "852dbcb8-3c2d-4622-beeb-3fc265a7f906"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:07.699140 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:07.699050 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/852dbcb8-3c2d-4622-beeb-3fc265a7f906-workload-socket" pod "852dbcb8-3c2d-4622-beeb-3fc265a7f906" (UID: "852dbcb8-3c2d-4622-beeb-3fc265a7f906"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:07.699140 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:07.698984 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/852dbcb8-3c2d-4622-beeb-3fc265a7f906-istiod-ca-cert" pod "852dbcb8-3c2d-4622-beeb-3fc265a7f906" (UID: "852dbcb8-3c2d-4622-beeb-3fc265a7f906"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:36:07.699445 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:07.699215 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/852dbcb8-3c2d-4622-beeb-3fc265a7f906-istio-data" pod "852dbcb8-3c2d-4622-beeb-3fc265a7f906" (UID: "852dbcb8-3c2d-4622-beeb-3fc265a7f906"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:07.699445 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:07.699312 6720 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/852dbcb8-3c2d-4622-beeb-3fc265a7f906-istio-data\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:07.699445 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:07.699333 6720 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/852dbcb8-3c2d-4622-beeb-3fc265a7f906-istiod-ca-cert\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:07.699445 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:07.699350 6720 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/852dbcb8-3c2d-4622-beeb-3fc265a7f906-credential-socket\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:07.699445 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:07.699364 6720 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/852dbcb8-3c2d-4622-beeb-3fc265a7f906-workload-certs\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:07.699445 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:07.699379 6720 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/852dbcb8-3c2d-4622-beeb-3fc265a7f906-workload-socket\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:07.701181 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:07.701155 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/852dbcb8-3c2d-4622-beeb-3fc265a7f906-istio-podinfo" pod "852dbcb8-3c2d-4622-beeb-3fc265a7f906" (UID: "852dbcb8-3c2d-4622-beeb-3fc265a7f906"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:36:07.701296 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:07.701179 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/852dbcb8-3c2d-4622-beeb-3fc265a7f906-istio-envoy" pod "852dbcb8-3c2d-4622-beeb-3fc265a7f906" (UID: "852dbcb8-3c2d-4622-beeb-3fc265a7f906"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:07.701508 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:07.701487 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/852dbcb8-3c2d-4622-beeb-3fc265a7f906-istio-token" pod "852dbcb8-3c2d-4622-beeb-3fc265a7f906" (UID: "852dbcb8-3c2d-4622-beeb-3fc265a7f906"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:36:07.701508 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:07.701495 6720 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/852dbcb8-3c2d-4622-beeb-3fc265a7f906-kube-api-access-v2lf4" pod "852dbcb8-3c2d-4622-beeb-3fc265a7f906" (UID: "852dbcb8-3c2d-4622-beeb-3fc265a7f906"). InnerVolumeSpecName "kube-api-access-v2lf4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:36:07.800691 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:07.800654 6720 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/852dbcb8-3c2d-4622-beeb-3fc265a7f906-istio-token\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:07.800691 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:07.800684 6720 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/852dbcb8-3c2d-4622-beeb-3fc265a7f906-istio-podinfo\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:07.800691 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:07.800695 6720 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2lf4\" (UniqueName: \"kubernetes.io/projected/852dbcb8-3c2d-4622-beeb-3fc265a7f906-kube-api-access-v2lf4\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:07.800951 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:07.800703 6720 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/852dbcb8-3c2d-4622-beeb-3fc265a7f906-istio-envoy\") on node \"ip-10-0-137-181.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:08.064834 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:08.064800 6720 generic.go:356] "Generic (PLEG): container finished" podID="852dbcb8-3c2d-4622-beeb-3fc265a7f906" containerID="dffc857ed3bc39215ce2a2834d0d81edb8861fe1f13dc1b926ef70b2246d4ae5" exitCode=0 Oct 30 22:36:08.065008 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:08.064847 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-nv8nz" event={"ID":"852dbcb8-3c2d-4622-beeb-3fc265a7f906","Type":"ContainerDied","Data":"dffc857ed3bc39215ce2a2834d0d81edb8861fe1f13dc1b926ef70b2246d4ae5"} Oct 30 22:36:08.065008 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:08.064856 6720 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-nv8nz" Oct 30 22:36:08.065008 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:08.064889 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-nv8nz" event={"ID":"852dbcb8-3c2d-4622-beeb-3fc265a7f906","Type":"ContainerDied","Data":"4a39a29a8f3e597f3a7215dd79770ee23e01297f6f915102cfdbf1c11717653b"} Oct 30 22:36:08.065008 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:08.064910 6720 scope.go:122] "RemoveContainer" containerID="dffc857ed3bc39215ce2a2834d0d81edb8861fe1f13dc1b926ef70b2246d4ae5" Oct 30 22:36:08.073725 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:08.073707 6720 scope.go:122] "RemoveContainer" containerID="dffc857ed3bc39215ce2a2834d0d81edb8861fe1f13dc1b926ef70b2246d4ae5" Oct 30 22:36:08.074016 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:36:08.073992 6720 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dffc857ed3bc39215ce2a2834d0d81edb8861fe1f13dc1b926ef70b2246d4ae5\": container with ID starting with dffc857ed3bc39215ce2a2834d0d81edb8861fe1f13dc1b926ef70b2246d4ae5 not found: ID does not exist" containerID="dffc857ed3bc39215ce2a2834d0d81edb8861fe1f13dc1b926ef70b2246d4ae5" Oct 30 22:36:08.074102 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:08.074026 6720 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dffc857ed3bc39215ce2a2834d0d81edb8861fe1f13dc1b926ef70b2246d4ae5"} err="failed to get container status \"dffc857ed3bc39215ce2a2834d0d81edb8861fe1f13dc1b926ef70b2246d4ae5\": rpc error: code = NotFound desc = could not find container \"dffc857ed3bc39215ce2a2834d0d81edb8861fe1f13dc1b926ef70b2246d4ae5\": container with ID starting with dffc857ed3bc39215ce2a2834d0d81edb8861fe1f13dc1b926ef70b2246d4ae5 not found: ID does not exist" Oct 30 22:36:08.085155 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:08.085129 6720 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-nv8nz"] Oct 30 22:36:08.091363 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:08.091308 6720 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-939a16-openshift-default-5fccf878b7-nv8nz"] Oct 30 22:36:08.108743 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:08.108719 6720 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-87c457c84-q9js5_bd0b4436-451d-4abd-aba0-3ebb841ba48e/authorino/0.log" Oct 30 22:36:09.954625 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:09.954587 6720 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="852dbcb8-3c2d-4622-beeb-3fc265a7f906" path="/var/lib/kubelet/pods/852dbcb8-3c2d-4622-beeb-3fc265a7f906/volumes" Oct 30 22:36:10.403534 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:36:10.403498 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 22:36:11.806849 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:11.806820 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:36:11.969199 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:11.969170 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:36:13.520960 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:13.520873 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:36:18.935799 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:18.935765 6720 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-cptxw_9a04fd45-c758-4a59-8c04-c9499862dcbd/manager/0.log" Oct 30 22:36:19.322742 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:19.322662 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:36:19.332873 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:19.332839 6720 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-phjxr_753e9e9a-5c68-4540-8fc7-858514ca428e/manager/0.log" Oct 30 22:36:19.863458 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:19.863417 6720 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-57d6c9d545-ntxm7_91b3aaaf-cf84-44d0-9156-7174e41f8bed/manager/0.log" Oct 30 22:36:20.044115 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:20.044089 6720 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-7c7vd_0cefd95b-2c8c-4271-93b3-4d37e6e00d8e/server/0.log" Oct 30 22:36:20.223293 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:20.223215 6720 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-ckx8d_838cabb6-566e-4376-8c49-f0801ac171fc/manager/0.log" Oct 30 22:36:23.265448 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:23.265418 6720 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-dgbsj_36ab689f-07ab-4d9a-b02b-01a0c2c468d9/postgres/0.log" Oct 30 22:36:24.783362 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:24.783334 6720 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-87c457c84-q9js5_bd0b4436-451d-4abd-aba0-3ebb841ba48e/authorino/0.log" Oct 30 22:36:26.015987 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:26.015954 6720 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-x5rxl_1bb7e9b5-60a0-4cf9-a5ab-2e5e76d077aa/manager/0.log" Oct 30 22:36:27.344206 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:27.344175 6720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-rmpbr_47ca938c-cc9e-45b3-95c1-1f9cf1206316/discovery/0.log" Oct 30 22:36:27.522653 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:27.522624 6720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-78559bccc4-9vqg5_0271c996-234d-4291-aa7a-77d43569c247/kube-auth-proxy/0.log" Oct 30 22:36:28.096666 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:28.096640 6720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-85544d4cd7-8grjn_b3dfec03-d265-405d-bc13-fcf8f44b794d/istio-proxy/0.log" Oct 30 22:36:28.867904 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:28.867873 6720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-85544d4cd7-sqg22_ccc62572-e0cd-43fb-8fc2-7715f2223da3/istio-proxy/0.log" Oct 30 22:36:30.631590 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:30.631537 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:36:40.400913 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:36:40.400874 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 22:36:43.096863 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:43.096831 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:36:43.246465 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:43.246433 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:36:48.019049 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:48.019016 6720 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-vckqz_17127db1-cf66-4951-ba5d-eb486ee9d3d5/global-pull-secret-syncer/0.log" Oct 30 22:36:48.218686 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:48.218656 6720 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" Oct 30 22:36:48.529276 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:48.529246 6720 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-181.us-east-2.compute.internal_5fb540c197b86dcf7fd282ae878b96ce/haproxy/0.log" Oct 30 22:36:56.025507 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:56.025475 6720 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-87c457c84-q9js5_bd0b4436-451d-4abd-aba0-3ebb841ba48e/authorino/0.log" Oct 30 22:36:56.754509 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:56.754475 6720 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-x5rxl_1bb7e9b5-60a0-4cf9-a5ab-2e5e76d077aa/manager/0.log" Oct 30 22:36:58.526357 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:58.526329 6720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-4xbzf_f4b104cb-8d0c-4519-86e0-07ea0e71a4a0/kube-state-metrics/0.log" Oct 30 22:36:58.552581 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:58.552553 6720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-4xbzf_f4b104cb-8d0c-4519-86e0-07ea0e71a4a0/kube-rbac-proxy-main/0.log" Oct 30 22:36:58.581798 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:58.581770 6720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-4xbzf_f4b104cb-8d0c-4519-86e0-07ea0e71a4a0/kube-rbac-proxy-self/0.log" Oct 30 22:36:58.617704 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:58.617677 6720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-c74454f78-xph4m_df5bcaa5-7723-4ea4-bbc6-dbf65d3ecb2d/metrics-server/0.log" Oct 30 22:36:58.650952 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:58.650913 6720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-qv74l_a2e2c1af-8ab2-49c6-91ac-4712b88e0571/monitoring-plugin/0.log" Oct 30 22:36:58.689985 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:58.689960 6720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hbg8k_c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62/node-exporter/0.log" Oct 30 22:36:58.721136 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:58.721108 6720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hbg8k_c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62/kube-rbac-proxy/0.log" Oct 30 22:36:58.760453 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:58.760430 6720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hbg8k_c7cdfe5f-5f27-41a4-87de-7f5f6d2c1d62/init-textfile/0.log" Oct 30 22:36:59.055096 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:59.055065 6720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-44hwx_971a083e-69b8-46d1-b56a-9ccf0f4d557d/kube-rbac-proxy-main/0.log" Oct 30 22:36:59.088462 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:59.088431 6720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-44hwx_971a083e-69b8-46d1-b56a-9ccf0f4d557d/kube-rbac-proxy-self/0.log" Oct 30 22:36:59.121156 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:59.121130 6720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-44hwx_971a083e-69b8-46d1-b56a-9ccf0f4d557d/openshift-state-metrics/0.log" Oct 30 22:36:59.164821 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:59.164793 6720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_163b57d1-c2a7-4939-8f82-77c01cd1721b/prometheus/0.log" Oct 30 22:36:59.199051 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:59.199023 6720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_163b57d1-c2a7-4939-8f82-77c01cd1721b/config-reloader/0.log" Oct 30 22:36:59.236984 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:59.236956 6720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_163b57d1-c2a7-4939-8f82-77c01cd1721b/thanos-sidecar/0.log" Oct 30 22:36:59.267627 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:59.267604 6720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_163b57d1-c2a7-4939-8f82-77c01cd1721b/kube-rbac-proxy-web/0.log" Oct 30 22:36:59.297877 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:59.297852 6720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_163b57d1-c2a7-4939-8f82-77c01cd1721b/kube-rbac-proxy/0.log" Oct 30 22:36:59.329437 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:59.329355 6720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_163b57d1-c2a7-4939-8f82-77c01cd1721b/kube-rbac-proxy-thanos/0.log" Oct 30 22:36:59.360141 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:59.360117 6720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_163b57d1-c2a7-4939-8f82-77c01cd1721b/init-config-reloader/0.log" Oct 30 22:36:59.398399 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:59.398370 6720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-hrkzk_568ba29e-ccc3-4a4c-9a4f-192c0b931b0d/prometheus-operator/0.log" Oct 30 22:36:59.424680 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:59.424658 6720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-hrkzk_568ba29e-ccc3-4a4c-9a4f-192c0b931b0d/kube-rbac-proxy/0.log" Oct 30 22:36:59.613791 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:59.613769 6720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6fd8fdc4c5-xlslr_90136025-abf7-4dc6-ac4b-c3050fc417f6/thanos-query/0.log" Oct 30 22:36:59.647151 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:59.647127 6720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6fd8fdc4c5-xlslr_90136025-abf7-4dc6-ac4b-c3050fc417f6/kube-rbac-proxy-web/0.log" Oct 30 22:36:59.680908 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:59.680881 6720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6fd8fdc4c5-xlslr_90136025-abf7-4dc6-ac4b-c3050fc417f6/kube-rbac-proxy/0.log" Oct 30 22:36:59.713397 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:59.713362 6720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6fd8fdc4c5-xlslr_90136025-abf7-4dc6-ac4b-c3050fc417f6/prom-label-proxy/0.log" Oct 30 22:36:59.746207 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:59.746186 6720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6fd8fdc4c5-xlslr_90136025-abf7-4dc6-ac4b-c3050fc417f6/kube-rbac-proxy-rules/0.log" Oct 30 22:36:59.776524 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:36:59.776506 6720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6fd8fdc4c5-xlslr_90136025-abf7-4dc6-ac4b-c3050fc417f6/kube-rbac-proxy-metrics/0.log" Oct 30 22:37:00.221258 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.221220 6720 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-k2j9d/perf-node-gather-daemonset-vz579"] Oct 30 22:37:00.221632 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.221618 6720 cpu_manager.go:414] "containerMap: removing container" podUID="6b6fc7b5-8eca-4f2c-b012-6f9375211314" containerName="maas-api" Oct 30 22:37:00.221694 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.221633 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b6fc7b5-8eca-4f2c-b012-6f9375211314" containerName="maas-api" Oct 30 22:37:00.221694 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.221649 6720 cpu_manager.go:414] "containerMap: removing container" podUID="5ae33413-cc17-4bbb-ac91-ec406f02c125" containerName="main" Oct 30 22:37:00.221694 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.221654 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ae33413-cc17-4bbb-ac91-ec406f02c125" containerName="main" Oct 30 22:37:00.221694 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.221665 6720 cpu_manager.go:414] "containerMap: removing container" podUID="8566ec43-9f32-4ad1-a46f-fc94ccf1b24d" containerName="istio-proxy" Oct 30 22:37:00.221694 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.221672 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8566ec43-9f32-4ad1-a46f-fc94ccf1b24d" containerName="istio-proxy" Oct 30 22:37:00.221694 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.221681 6720 cpu_manager.go:414] "containerMap: removing container" podUID="1f53144d-ae7f-493c-ac5b-714b9ad3b29e" containerName="maas-api" Oct 30 22:37:00.221694 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.221688 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f53144d-ae7f-493c-ac5b-714b9ad3b29e" containerName="maas-api" Oct 30 22:37:00.221969 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.221704 6720 cpu_manager.go:414] "containerMap: removing container" podUID="5267ee0d-42b1-482f-aa8c-f1cba3b748ea" containerName="maas-api" Oct 30 22:37:00.221969 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.221712 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5267ee0d-42b1-482f-aa8c-f1cba3b748ea" containerName="maas-api" Oct 30 22:37:00.221969 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.221721 6720 cpu_manager.go:414] "containerMap: removing container" podUID="24a24a49-0696-4e41-8b93-0b11814f98cd" containerName="maas-api" Oct 30 22:37:00.221969 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.221729 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24a24a49-0696-4e41-8b93-0b11814f98cd" containerName="maas-api" Oct 30 22:37:00.221969 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.221739 6720 cpu_manager.go:414] "containerMap: removing container" podUID="3360dc0f-497d-4211-a73f-c5ac0ff8f204" containerName="main" Oct 30 22:37:00.221969 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.221745 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3360dc0f-497d-4211-a73f-c5ac0ff8f204" containerName="main" Oct 30 22:37:00.221969 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.221762 6720 cpu_manager.go:414] "containerMap: removing container" podUID="2f501cad-6ca4-4615-b978-bd05ee0cfa2c" containerName="maas-api" Oct 30 22:37:00.221969 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.221769 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f501cad-6ca4-4615-b978-bd05ee0cfa2c" containerName="maas-api" Oct 30 22:37:00.221969 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.221778 6720 cpu_manager.go:414] "containerMap: removing container" podUID="d065e788-c972-4683-84ed-86a70ed4791d" containerName="main" Oct 30 22:37:00.221969 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.221786 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d065e788-c972-4683-84ed-86a70ed4791d" containerName="main" Oct 30 22:37:00.221969 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.221796 6720 cpu_manager.go:414] "containerMap: removing container" podUID="97e1e65d-1d50-4097-98ca-2944fa182807" containerName="main" Oct 30 22:37:00.221969 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.221803 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97e1e65d-1d50-4097-98ca-2944fa182807" containerName="main" Oct 30 22:37:00.221969 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.221814 6720 cpu_manager.go:414] "containerMap: removing container" podUID="0e1f3f99-8520-4097-b948-3ca747b75ec9" containerName="maas-api" Oct 30 22:37:00.221969 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.221821 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e1f3f99-8520-4097-b948-3ca747b75ec9" containerName="maas-api" Oct 30 22:37:00.221969 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.221831 6720 cpu_manager.go:414] "containerMap: removing container" podUID="961c219a-a80f-46eb-ab6c-2920a28419e2" containerName="istio-proxy" Oct 30 22:37:00.221969 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.221836 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="961c219a-a80f-46eb-ab6c-2920a28419e2" containerName="istio-proxy" Oct 30 22:37:00.221969 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.221843 6720 cpu_manager.go:414] "containerMap: removing container" podUID="083d25b4-5991-4298-b2f3-392d28c4e620" containerName="maas-api" Oct 30 22:37:00.221969 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.221847 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="083d25b4-5991-4298-b2f3-392d28c4e620" containerName="maas-api" Oct 30 22:37:00.221969 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.221853 6720 cpu_manager.go:414] "containerMap: removing container" podUID="c9cfa5a4-4d17-41f4-985c-c119257e3720" containerName="maas-api" Oct 30 22:37:00.221969 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.221857 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9cfa5a4-4d17-41f4-985c-c119257e3720" containerName="maas-api" Oct 30 22:37:00.221969 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.221867 6720 cpu_manager.go:414] "containerMap: removing container" podUID="423a4a57-1e84-4833-ac24-261dcad52016" containerName="maas-api" Oct 30 22:37:00.221969 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.221873 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="423a4a57-1e84-4833-ac24-261dcad52016" containerName="maas-api" Oct 30 22:37:00.221969 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.221884 6720 cpu_manager.go:414] "containerMap: removing container" podUID="9bb6805f-f355-4ea8-80b7-542215c42022" containerName="main" Oct 30 22:37:00.221969 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.221893 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bb6805f-f355-4ea8-80b7-542215c42022" containerName="main" Oct 30 22:37:00.221969 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.221903 6720 cpu_manager.go:414] "containerMap: removing container" podUID="f1b6e92c-2952-4bcb-ae33-91e7271342fe" containerName="main" Oct 30 22:37:00.221969 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.221910 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1b6e92c-2952-4bcb-ae33-91e7271342fe" containerName="main" Oct 30 22:37:00.221969 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.221920 6720 cpu_manager.go:414] "containerMap: removing container" podUID="61ff25c3-567e-4fef-ad01-057108004032" containerName="istio-proxy" Oct 30 22:37:00.221969 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.221924 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61ff25c3-567e-4fef-ad01-057108004032" containerName="istio-proxy" Oct 30 22:37:00.221969 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.221929 6720 cpu_manager.go:414] "containerMap: removing container" podUID="9c5ce4b5-7597-47c4-9530-56544816a191" containerName="maas-api" Oct 30 22:37:00.221969 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.221934 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c5ce4b5-7597-47c4-9530-56544816a191" containerName="maas-api" Oct 30 22:37:00.221969 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.221946 6720 cpu_manager.go:414] "containerMap: removing container" podUID="257edb08-2e86-4f49-be57-848ace5958b1" containerName="main" Oct 30 22:37:00.221969 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.221950 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="257edb08-2e86-4f49-be57-848ace5958b1" containerName="main" Oct 30 22:37:00.221969 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.221956 6720 cpu_manager.go:414] "containerMap: removing container" podUID="85316c69-7ebd-4fb6-9195-44d1fba2dfbc" containerName="main" Oct 30 22:37:00.221969 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.221959 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85316c69-7ebd-4fb6-9195-44d1fba2dfbc" containerName="main" Oct 30 22:37:00.221969 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.221967 6720 cpu_manager.go:414] "containerMap: removing container" podUID="09662f19-1ef9-406c-85ef-e402505875b4" containerName="main" Oct 30 22:37:00.221969 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.221971 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09662f19-1ef9-406c-85ef-e402505875b4" containerName="main" Oct 30 22:37:00.221969 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.221977 6720 cpu_manager.go:414] "containerMap: removing container" podUID="e7f48858-397b-46fe-84b4-f19ef4afa227" containerName="main" Oct 30 22:37:00.221969 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.221981 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7f48858-397b-46fe-84b4-f19ef4afa227" containerName="main" Oct 30 22:37:00.223561 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.221990 6720 cpu_manager.go:414] "containerMap: removing container" podUID="a771bf56-fa60-4bac-bfd3-4aa0ccf2bf82" containerName="maas-api" Oct 30 22:37:00.223561 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.221994 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a771bf56-fa60-4bac-bfd3-4aa0ccf2bf82" containerName="maas-api" Oct 30 22:37:00.223561 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.222000 6720 cpu_manager.go:414] "containerMap: removing container" podUID="e12f67b4-89de-4975-816d-f44fff4763f9" containerName="maas-api" Oct 30 22:37:00.223561 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.222004 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e12f67b4-89de-4975-816d-f44fff4763f9" containerName="maas-api" Oct 30 22:37:00.223561 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.222012 6720 cpu_manager.go:414] "containerMap: removing container" podUID="22c4c993-7bb7-492e-a4bc-05e7fa6c7b7f" containerName="maas-api" Oct 30 22:37:00.223561 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.222016 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22c4c993-7bb7-492e-a4bc-05e7fa6c7b7f" containerName="maas-api" Oct 30 22:37:00.223561 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.222022 6720 cpu_manager.go:414] "containerMap: removing container" podUID="b6c2aa19-cad5-435a-a573-56146651be06" containerName="maas-api" Oct 30 22:37:00.223561 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.222026 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6c2aa19-cad5-435a-a573-56146651be06" containerName="maas-api" Oct 30 22:37:00.223561 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.222032 6720 cpu_manager.go:414] "containerMap: removing container" podUID="1ea7cd43-6136-4a44-921d-65fbf8c39f76" containerName="istio-proxy" Oct 30 22:37:00.223561 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.222036 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ea7cd43-6136-4a44-921d-65fbf8c39f76" containerName="istio-proxy" Oct 30 22:37:00.223561 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.222047 6720 cpu_manager.go:414] "containerMap: removing container" podUID="457d2f97-63ab-41f4-b761-294d0ba3045e" containerName="main" Oct 30 22:37:00.223561 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.222051 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="457d2f97-63ab-41f4-b761-294d0ba3045e" containerName="main" Oct 30 22:37:00.223561 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.222057 6720 cpu_manager.go:414] "containerMap: removing container" podUID="852dbcb8-3c2d-4622-beeb-3fc265a7f906" containerName="istio-proxy" Oct 30 22:37:00.223561 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.222061 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="852dbcb8-3c2d-4622-beeb-3fc265a7f906" containerName="istio-proxy" Oct 30 22:37:00.223561 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.222069 6720 cpu_manager.go:414] "containerMap: removing container" podUID="d31cf31e-58d7-4ace-8959-6cba1429daec" containerName="istio-proxy" Oct 30 22:37:00.223561 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.222074 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d31cf31e-58d7-4ace-8959-6cba1429daec" containerName="istio-proxy" Oct 30 22:37:00.223561 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.222080 6720 cpu_manager.go:414] "containerMap: removing container" podUID="fc5329cd-398f-42a3-8e4d-82d3b6446773" containerName="istio-proxy" Oct 30 22:37:00.223561 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.222084 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc5329cd-398f-42a3-8e4d-82d3b6446773" containerName="istio-proxy" Oct 30 22:37:00.223561 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.222089 6720 cpu_manager.go:414] "containerMap: removing container" podUID="883a9007-3704-47d0-b59e-8591e1d33186" containerName="maas-api" Oct 30 22:37:00.223561 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.222093 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="883a9007-3704-47d0-b59e-8591e1d33186" containerName="maas-api" Oct 30 22:37:00.223561 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.222100 6720 cpu_manager.go:414] "containerMap: removing container" podUID="cc1b52ec-a38c-409e-810c-7ae3998c2ebd" containerName="istio-proxy" Oct 30 22:37:00.223561 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.222104 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc1b52ec-a38c-409e-810c-7ae3998c2ebd" containerName="istio-proxy" Oct 30 22:37:00.223561 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.222111 6720 cpu_manager.go:414] "containerMap: removing container" podUID="132ce48a-6190-4f71-8b77-1a28c917030d" containerName="maas-api" Oct 30 22:37:00.223561 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.222115 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="132ce48a-6190-4f71-8b77-1a28c917030d" containerName="maas-api" Oct 30 22:37:00.223561 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.222121 6720 cpu_manager.go:414] "containerMap: removing container" podUID="744e223f-d9d1-45c3-9f37-b4d29091852c" containerName="istio-proxy" Oct 30 22:37:00.223561 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.222126 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="744e223f-d9d1-45c3-9f37-b4d29091852c" containerName="istio-proxy" Oct 30 22:37:00.223561 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.222134 6720 cpu_manager.go:414] "containerMap: removing container" podUID="9200859d-b207-4764-a75c-270ba29b4155" containerName="istio-proxy" Oct 30 22:37:00.223561 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.222138 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9200859d-b207-4764-a75c-270ba29b4155" containerName="istio-proxy" Oct 30 22:37:00.223561 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.222145 6720 cpu_manager.go:414] "containerMap: removing container" podUID="47d2f169-c1d8-4533-be16-9d195a7f73c9" containerName="istio-proxy" Oct 30 22:37:00.223561 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.222149 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47d2f169-c1d8-4533-be16-9d195a7f73c9" containerName="istio-proxy" Oct 30 22:37:00.223561 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.222158 6720 cpu_manager.go:414] "containerMap: removing container" podUID="ea5efed8-2dea-46bf-9162-55fe6a1459d5" containerName="istio-proxy" Oct 30 22:37:00.223561 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.222163 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea5efed8-2dea-46bf-9162-55fe6a1459d5" containerName="istio-proxy" Oct 30 22:37:00.223561 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.222169 6720 cpu_manager.go:414] "containerMap: removing container" podUID="919d285e-2e4f-41b9-95f8-87144df6a112" containerName="istio-proxy" Oct 30 22:37:00.223561 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.222173 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="919d285e-2e4f-41b9-95f8-87144df6a112" containerName="istio-proxy" Oct 30 22:37:00.223561 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.222179 6720 cpu_manager.go:414] "containerMap: removing container" podUID="f51bd7b2-4d7e-42af-9dbc-4083db247bf4" containerName="maas-api" Oct 30 22:37:00.223561 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.222184 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f51bd7b2-4d7e-42af-9dbc-4083db247bf4" containerName="maas-api" Oct 30 22:37:00.223561 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.222190 6720 cpu_manager.go:414] "containerMap: removing container" podUID="7365fe23-d73c-467d-bd8b-d0177976452f" containerName="maas-api" Oct 30 22:37:00.223561 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.222194 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7365fe23-d73c-467d-bd8b-d0177976452f" containerName="maas-api" Oct 30 22:37:00.223561 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.222199 6720 cpu_manager.go:414] "containerMap: removing container" podUID="22545d04-38b5-4d64-89a0-9e533875220e" containerName="maas-api" Oct 30 22:37:00.223561 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.222203 6720 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22545d04-38b5-4d64-89a0-9e533875220e" containerName="maas-api" Oct 30 22:37:00.223561 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.222282 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f501cad-6ca4-4615-b978-bd05ee0cfa2c" containerName="maas-api" Oct 30 22:37:00.223561 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.222291 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="d065e788-c972-4683-84ed-86a70ed4791d" containerName="main" Oct 30 22:37:00.223561 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.222296 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="97e1e65d-1d50-4097-98ca-2944fa182807" containerName="main" Oct 30 22:37:00.223561 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.222304 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e1f3f99-8520-4097-b948-3ca747b75ec9" containerName="maas-api" Oct 30 22:37:00.223561 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.222309 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="961c219a-a80f-46eb-ab6c-2920a28419e2" containerName="istio-proxy" Oct 30 22:37:00.223561 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.222316 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="083d25b4-5991-4298-b2f3-392d28c4e620" containerName="maas-api" Oct 30 22:37:00.223561 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.222321 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9cfa5a4-4d17-41f4-985c-c119257e3720" containerName="maas-api" Oct 30 22:37:00.223561 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.222326 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="423a4a57-1e84-4833-ac24-261dcad52016" containerName="maas-api" Oct 30 22:37:00.223561 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.222333 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bb6805f-f355-4ea8-80b7-542215c42022" containerName="main" Oct 30 22:37:00.223561 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.222339 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1b6e92c-2952-4bcb-ae33-91e7271342fe" containerName="main" Oct 30 22:37:00.223561 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.222344 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="61ff25c3-567e-4fef-ad01-057108004032" containerName="istio-proxy" Oct 30 22:37:00.223561 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.222348 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c5ce4b5-7597-47c4-9530-56544816a191" containerName="maas-api" Oct 30 22:37:00.223561 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.222354 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="257edb08-2e86-4f49-be57-848ace5958b1" containerName="main" Oct 30 22:37:00.223561 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.222358 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="85316c69-7ebd-4fb6-9195-44d1fba2dfbc" containerName="main" Oct 30 22:37:00.223561 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.222363 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="09662f19-1ef9-406c-85ef-e402505875b4" containerName="main" Oct 30 22:37:00.223561 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.222368 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7f48858-397b-46fe-84b4-f19ef4afa227" containerName="main" Oct 30 22:37:00.223561 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.222373 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="a771bf56-fa60-4bac-bfd3-4aa0ccf2bf82" containerName="maas-api" Oct 30 22:37:00.223561 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.222377 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="e12f67b4-89de-4975-816d-f44fff4763f9" containerName="maas-api" Oct 30 22:37:00.223561 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.222397 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="22c4c993-7bb7-492e-a4bc-05e7fa6c7b7f" containerName="maas-api" Oct 30 22:37:00.223561 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.222402 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6c2aa19-cad5-435a-a573-56146651be06" containerName="maas-api" Oct 30 22:37:00.223561 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.222408 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ea7cd43-6136-4a44-921d-65fbf8c39f76" containerName="istio-proxy" Oct 30 22:37:00.223561 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.222414 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="457d2f97-63ab-41f4-b761-294d0ba3045e" containerName="main" Oct 30 22:37:00.223561 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.222420 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="852dbcb8-3c2d-4622-beeb-3fc265a7f906" containerName="istio-proxy" Oct 30 22:37:00.223561 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.222426 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="d31cf31e-58d7-4ace-8959-6cba1429daec" containerName="istio-proxy" Oct 30 22:37:00.223561 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.222430 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc5329cd-398f-42a3-8e4d-82d3b6446773" containerName="istio-proxy" Oct 30 22:37:00.223561 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.222436 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="883a9007-3704-47d0-b59e-8591e1d33186" containerName="maas-api" Oct 30 22:37:00.223561 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.222440 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc1b52ec-a38c-409e-810c-7ae3998c2ebd" containerName="istio-proxy" Oct 30 22:37:00.223561 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.222446 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="132ce48a-6190-4f71-8b77-1a28c917030d" containerName="maas-api" Oct 30 22:37:00.223561 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.222451 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="744e223f-d9d1-45c3-9f37-b4d29091852c" containerName="istio-proxy" Oct 30 22:37:00.223561 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.222456 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="9200859d-b207-4764-a75c-270ba29b4155" containerName="istio-proxy" Oct 30 22:37:00.223561 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.222462 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="47d2f169-c1d8-4533-be16-9d195a7f73c9" containerName="istio-proxy" Oct 30 22:37:00.223561 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.222468 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea5efed8-2dea-46bf-9162-55fe6a1459d5" containerName="istio-proxy" Oct 30 22:37:00.223561 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.222474 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="919d285e-2e4f-41b9-95f8-87144df6a112" containerName="istio-proxy" Oct 30 22:37:00.223561 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.222479 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="f51bd7b2-4d7e-42af-9dbc-4083db247bf4" containerName="maas-api" Oct 30 22:37:00.223561 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.222483 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="7365fe23-d73c-467d-bd8b-d0177976452f" containerName="maas-api" Oct 30 22:37:00.223561 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.222488 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="22545d04-38b5-4d64-89a0-9e533875220e" containerName="maas-api" Oct 30 22:37:00.223561 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.222493 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b6fc7b5-8eca-4f2c-b012-6f9375211314" containerName="maas-api" Oct 30 22:37:00.223561 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.222498 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ae33413-cc17-4bbb-ac91-ec406f02c125" containerName="main" Oct 30 22:37:00.223561 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.222503 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="8566ec43-9f32-4ad1-a46f-fc94ccf1b24d" containerName="istio-proxy" Oct 30 22:37:00.223561 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.222507 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f53144d-ae7f-493c-ac5b-714b9ad3b29e" containerName="maas-api" Oct 30 22:37:00.223561 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.222513 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="5267ee0d-42b1-482f-aa8c-f1cba3b748ea" containerName="maas-api" Oct 30 22:37:00.223561 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.222519 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="24a24a49-0696-4e41-8b93-0b11814f98cd" containerName="maas-api" Oct 30 22:37:00.223561 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.222524 6720 memory_manager.go:365] "RemoveStaleState removing state" podUID="3360dc0f-497d-4211-a73f-c5ac0ff8f204" containerName="main" Oct 30 22:37:00.227008 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.226990 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-k2j9d/perf-node-gather-daemonset-vz579" Oct 30 22:37:00.234793 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.234708 6720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-k2j9d\"/\"kube-root-ca.crt\"" Oct 30 22:37:00.235847 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.235822 6720 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-k2j9d\"/\"openshift-service-ca.crt\"" Oct 30 22:37:00.235982 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.235912 6720 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-k2j9d\"/\"default-dockercfg-xhph6\"" Oct 30 22:37:00.237234 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.237215 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-k2j9d/perf-node-gather-daemonset-vz579"] Oct 30 22:37:00.394751 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.394712 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0e3bd016-e4ef-44af-9c84-94fd1cde7024-proc\") pod \"perf-node-gather-daemonset-vz579\" (UID: \"0e3bd016-e4ef-44af-9c84-94fd1cde7024\") " pod="openshift-must-gather-k2j9d/perf-node-gather-daemonset-vz579" Oct 30 22:37:00.394945 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.394764 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kslrh\" (UniqueName: \"kubernetes.io/projected/0e3bd016-e4ef-44af-9c84-94fd1cde7024-kube-api-access-kslrh\") pod \"perf-node-gather-daemonset-vz579\" (UID: \"0e3bd016-e4ef-44af-9c84-94fd1cde7024\") " pod="openshift-must-gather-k2j9d/perf-node-gather-daemonset-vz579" Oct 30 22:37:00.394945 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.394888 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0e3bd016-e4ef-44af-9c84-94fd1cde7024-lib-modules\") pod \"perf-node-gather-daemonset-vz579\" (UID: \"0e3bd016-e4ef-44af-9c84-94fd1cde7024\") " pod="openshift-must-gather-k2j9d/perf-node-gather-daemonset-vz579" Oct 30 22:37:00.394945 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.394926 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0e3bd016-e4ef-44af-9c84-94fd1cde7024-podres\") pod \"perf-node-gather-daemonset-vz579\" (UID: \"0e3bd016-e4ef-44af-9c84-94fd1cde7024\") " pod="openshift-must-gather-k2j9d/perf-node-gather-daemonset-vz579" Oct 30 22:37:00.395091 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.394953 6720 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0e3bd016-e4ef-44af-9c84-94fd1cde7024-sys\") pod \"perf-node-gather-daemonset-vz579\" (UID: \"0e3bd016-e4ef-44af-9c84-94fd1cde7024\") " pod="openshift-must-gather-k2j9d/perf-node-gather-daemonset-vz579" Oct 30 22:37:00.495824 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.495748 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0e3bd016-e4ef-44af-9c84-94fd1cde7024-lib-modules\") pod \"perf-node-gather-daemonset-vz579\" (UID: \"0e3bd016-e4ef-44af-9c84-94fd1cde7024\") " pod="openshift-must-gather-k2j9d/perf-node-gather-daemonset-vz579" Oct 30 22:37:00.495824 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.495790 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0e3bd016-e4ef-44af-9c84-94fd1cde7024-podres\") pod \"perf-node-gather-daemonset-vz579\" (UID: \"0e3bd016-e4ef-44af-9c84-94fd1cde7024\") " pod="openshift-must-gather-k2j9d/perf-node-gather-daemonset-vz579" Oct 30 22:37:00.495824 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.495815 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0e3bd016-e4ef-44af-9c84-94fd1cde7024-sys\") pod \"perf-node-gather-daemonset-vz579\" (UID: \"0e3bd016-e4ef-44af-9c84-94fd1cde7024\") " pod="openshift-must-gather-k2j9d/perf-node-gather-daemonset-vz579" Oct 30 22:37:00.496060 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.495855 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0e3bd016-e4ef-44af-9c84-94fd1cde7024-proc\") pod \"perf-node-gather-daemonset-vz579\" (UID: \"0e3bd016-e4ef-44af-9c84-94fd1cde7024\") " pod="openshift-must-gather-k2j9d/perf-node-gather-daemonset-vz579" Oct 30 22:37:00.496060 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.495880 6720 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kslrh\" (UniqueName: \"kubernetes.io/projected/0e3bd016-e4ef-44af-9c84-94fd1cde7024-kube-api-access-kslrh\") pod \"perf-node-gather-daemonset-vz579\" (UID: \"0e3bd016-e4ef-44af-9c84-94fd1cde7024\") " pod="openshift-must-gather-k2j9d/perf-node-gather-daemonset-vz579" Oct 30 22:37:00.496060 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.495896 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0e3bd016-e4ef-44af-9c84-94fd1cde7024-podres\") pod \"perf-node-gather-daemonset-vz579\" (UID: \"0e3bd016-e4ef-44af-9c84-94fd1cde7024\") " pod="openshift-must-gather-k2j9d/perf-node-gather-daemonset-vz579" Oct 30 22:37:00.496060 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.495896 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0e3bd016-e4ef-44af-9c84-94fd1cde7024-lib-modules\") pod \"perf-node-gather-daemonset-vz579\" (UID: \"0e3bd016-e4ef-44af-9c84-94fd1cde7024\") " pod="openshift-must-gather-k2j9d/perf-node-gather-daemonset-vz579" Oct 30 22:37:00.496060 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.495953 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0e3bd016-e4ef-44af-9c84-94fd1cde7024-sys\") pod \"perf-node-gather-daemonset-vz579\" (UID: \"0e3bd016-e4ef-44af-9c84-94fd1cde7024\") " pod="openshift-must-gather-k2j9d/perf-node-gather-daemonset-vz579" Oct 30 22:37:00.496060 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.495945 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0e3bd016-e4ef-44af-9c84-94fd1cde7024-proc\") pod \"perf-node-gather-daemonset-vz579\" (UID: \"0e3bd016-e4ef-44af-9c84-94fd1cde7024\") " pod="openshift-must-gather-k2j9d/perf-node-gather-daemonset-vz579" Oct 30 22:37:00.509006 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.508981 6720 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kslrh\" (UniqueName: \"kubernetes.io/projected/0e3bd016-e4ef-44af-9c84-94fd1cde7024-kube-api-access-kslrh\") pod \"perf-node-gather-daemonset-vz579\" (UID: \"0e3bd016-e4ef-44af-9c84-94fd1cde7024\") " pod="openshift-must-gather-k2j9d/perf-node-gather-daemonset-vz579" Oct 30 22:37:00.540753 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.540731 6720 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-k2j9d/perf-node-gather-daemonset-vz579" Oct 30 22:37:00.821208 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:00.821184 6720 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-k2j9d/perf-node-gather-daemonset-vz579"] Oct 30 22:37:00.824125 ip-10-0-137-181 kubenswrapper[6720]: W1030 22:37:00.824091 6720 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0e3bd016_e4ef_44af_9c84_94fd1cde7024.slice/crio-8ff3f1fed790048b8a0f29fb13b31b9c6c29e1b21d816c0db184de2fa4a70a25 WatchSource:0}: Error finding container 8ff3f1fed790048b8a0f29fb13b31b9c6c29e1b21d816c0db184de2fa4a70a25: Status 404 returned error can't find the container with id 8ff3f1fed790048b8a0f29fb13b31b9c6c29e1b21d816c0db184de2fa4a70a25 Oct 30 22:37:01.319709 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:01.319670 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-k2j9d/perf-node-gather-daemonset-vz579" event={"ID":"0e3bd016-e4ef-44af-9c84-94fd1cde7024","Type":"ContainerStarted","Data":"e54d8168dbf95e47698ea15aaec9381bd712e1b20cb475996a92855e10c0699b"} Oct 30 22:37:01.319930 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:01.319735 6720 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-k2j9d/perf-node-gather-daemonset-vz579" event={"ID":"0e3bd016-e4ef-44af-9c84-94fd1cde7024","Type":"ContainerStarted","Data":"8ff3f1fed790048b8a0f29fb13b31b9c6c29e1b21d816c0db184de2fa4a70a25"} Oct 30 22:37:01.319930 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:01.319777 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-k2j9d/perf-node-gather-daemonset-vz579" Oct 30 22:37:01.340681 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:01.340627 6720 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-k2j9d/perf-node-gather-daemonset-vz579" podStartSLOduration=1.340610552 podStartE2EDuration="1.340610552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:37:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:37:01.340104301 +0000 UTC m=+3387.511069483" watchObservedRunningTime="2026-10-30 22:37:01.340610552 +0000 UTC m=+3387.511575734" Oct 30 22:37:02.298963 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:02.298898 6720 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-jg4wr_27f7c44f-0287-4608-a04a-7f108a25b121/konnectivity-agent/0.log" limit=1024 Oct 30 22:37:03.563557 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:03.563508 6720 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-cptxw_9a04fd45-c758-4a59-8c04-c9499862dcbd/manager/0.log" Oct 30 22:37:03.703344 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:03.703291 6720 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-cptxw_9a04fd45-c758-4a59-8c04-c9499862dcbd/manager/0.log" Oct 30 22:37:04.575886 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:04.575849 6720 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-phjxr_753e9e9a-5c68-4540-8fc7-858514ca428e/manager/0.log" Oct 30 22:37:04.676065 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:04.676022 6720 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-phjxr_753e9e9a-5c68-4540-8fc7-858514ca428e/manager/0.log" Oct 30 22:37:05.066507 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:05.066469 6720 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-57d6c9d545-ntxm7_91b3aaaf-cf84-44d0-9156-7174e41f8bed/manager/0.log" Oct 30 22:37:05.086315 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:05.086283 6720 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-7c7vd_0cefd95b-2c8c-4271-93b3-4d37e6e00d8e/server/0.log" Oct 30 22:37:05.136636 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:05.136607 6720 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-57d6c9d545-ntxm7_91b3aaaf-cf84-44d0-9156-7174e41f8bed/manager/0.log" Oct 30 22:37:05.159221 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:05.159195 6720 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-7c7vd_0cefd95b-2c8c-4271-93b3-4d37e6e00d8e/server/0.log" Oct 30 22:37:06.180026 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:06.179991 6720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-v9jlm_e2c4958f-c7a5-4b6a-9b29-c7e2cd7d2d41/dns/0.log" Oct 30 22:37:06.215488 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:06.215454 6720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-v9jlm_e2c4958f-c7a5-4b6a-9b29-c7e2cd7d2d41/kube-rbac-proxy/0.log" Oct 30 22:37:06.343192 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:06.343131 6720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-pnfw6_77670d00-a9cb-4607-bfb3-61ae91253cca/dns-node-resolver/0.log" Oct 30 22:37:06.503025 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:06.502919 6720 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-ckx8d_838cabb6-566e-4376-8c49-f0801ac171fc/manager/0.log" Oct 30 22:37:06.572398 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:06.572334 6720 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-ckx8d_838cabb6-566e-4376-8c49-f0801ac171fc/manager/0.log" Oct 30 22:37:07.335418 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:07.335367 6720 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-k2j9d/perf-node-gather-daemonset-vz579" Oct 30 22:37:07.371370 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:07.371328 6720 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-m7qjt_8b0ede8c-1c7c-479c-8c04-29d849fdbf43/node-ca/0.log" Oct 30 22:37:07.884074 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:07.884043 6720 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-cptxw_9a04fd45-c758-4a59-8c04-c9499862dcbd/manager/0.log" Oct 30 22:37:08.041755 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:08.041718 6720 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-cptxw_9a04fd45-c758-4a59-8c04-c9499862dcbd/manager/0.log" Oct 30 22:37:08.817553 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:08.817516 6720 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-phjxr_753e9e9a-5c68-4540-8fc7-858514ca428e/manager/0.log" Oct 30 22:37:08.999140 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:08.999085 6720 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-phjxr_753e9e9a-5c68-4540-8fc7-858514ca428e/manager/0.log" Oct 30 22:37:09.346458 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:09.346421 6720 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-57d6c9d545-ntxm7_91b3aaaf-cf84-44d0-9156-7174e41f8bed/manager/0.log" Oct 30 22:37:09.393112 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:09.393068 6720 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-7c7vd_0cefd95b-2c8c-4271-93b3-4d37e6e00d8e/server/0.log" Oct 30 22:37:09.521718 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:09.521682 6720 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-57d6c9d545-ntxm7_91b3aaaf-cf84-44d0-9156-7174e41f8bed/manager/0.log" Oct 30 22:37:09.566913 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:09.566878 6720 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-7c7vd_0cefd95b-2c8c-4271-93b3-4d37e6e00d8e/server/0.log" Oct 30 22:37:10.405694 ip-10-0-137-181 kubenswrapper[6720]: E1030 22:37:10.405649 6720 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-bhdkg" containerName="extractor" Oct 30 22:37:10.812115 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:10.812031 6720 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-ckx8d_838cabb6-566e-4376-8c49-f0801ac171fc/manager/0.log" Oct 30 22:37:10.930637 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:10.930600 6720 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-ckx8d_838cabb6-566e-4376-8c49-f0801ac171fc/manager/0.log" Oct 30 22:37:12.963894 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:12.963863 6720 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-cptxw_9a04fd45-c758-4a59-8c04-c9499862dcbd/manager/0.log" Oct 30 22:37:13.031432 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:13.031356 6720 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-cptxw_9a04fd45-c758-4a59-8c04-c9499862dcbd/manager/0.log" Oct 30 22:37:14.010702 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:14.010671 6720 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-phjxr_753e9e9a-5c68-4540-8fc7-858514ca428e/manager/0.log" Oct 30 22:37:14.100608 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:14.100580 6720 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-phjxr_753e9e9a-5c68-4540-8fc7-858514ca428e/manager/0.log" Oct 30 22:37:14.544532 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:14.544500 6720 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-57d6c9d545-ntxm7_91b3aaaf-cf84-44d0-9156-7174e41f8bed/manager/0.log" Oct 30 22:37:14.601047 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:14.601014 6720 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-7c7vd_0cefd95b-2c8c-4271-93b3-4d37e6e00d8e/server/0.log" Oct 30 22:37:14.611781 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:14.611750 6720 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-57d6c9d545-ntxm7_91b3aaaf-cf84-44d0-9156-7174e41f8bed/manager/0.log" Oct 30 22:37:14.682967 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:14.682934 6720 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-7c7vd_0cefd95b-2c8c-4271-93b3-4d37e6e00d8e/server/0.log" Oct 30 22:37:15.987410 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:15.987353 6720 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-ckx8d_838cabb6-566e-4376-8c49-f0801ac171fc/manager/0.log" Oct 30 22:37:16.060947 ip-10-0-137-181 kubenswrapper[6720]: I1030 22:37:16.060900 6720 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-ckx8d_838cabb6-566e-4376-8c49-f0801ac171fc/manager/0.log"